GitVita
New repositoryImport repositoryNew issue
Your repositories Your dashboard Sign in

microsoft/qdk

Public

mirrored fromhttps://github.com/microsoft/qdkAvailable

Watch0Fork0Star0
CodeCommitsIssuesPull requestsActionsInsightsSecurity
6f9284d127110b76c2edcad4a6f15aac9e9dc83a

Branches

Tags

  • No tags available.
0Branches0Tags
Go to file
Add file
Code

Clone

HTTPS

Download ZIP

qdk/fuzz

fuzz/README.md

420lines · modecode

RawDownload
Latest commit unavailable.
1# Fuzzing
2
3Based on [Fuzzing with cargo-fuzz](https://rust-fuzz.github.io/book/cargo-fuzz.html).
4
5For running locally you need the following steps.
6
7(**On Windows use [WSL](https://learn.microsoft.com/windows/wsl/).** Tested in WSL Ubuntu 22.04)
8
9## Prerequisites
10
11```bash
12rustup install nightly
13rustup default nightly
14
15cargo install cargo-fuzz
16```
17
18## Running
19
20**NOTE:** All the commands below are executed in WSL in the directory that contains this "fuzz" directory
21in a clean local copy of the repo (the whole repo is not built).
22
23```bash
24cargo fuzz list # Optional. See the available fuzzing targets.
25# compile # This fuzzing target fuzzes the `compile()` function.
26cargo fuzz run compile --features do_fuzz -- -seed_inputs=@fuzz/seed_inputs/compile/list.txt
27 # Build and run the fuzzing target "compile".
28# The build takes a few minutes. You may get an impression that the build takes place twice,
29# that is expected.
30# The run/fuzzing can last indefinitely without bumping into a panic. Stop with <Ctrl+c>.
31
32cargo fuzz run compile --features do_fuzz -- -help=1 # Optional. See the available run settings.
33
34# Optional. Run fuzzing for 10 runs at most, 5 seconds at most, generate ASCII-only fuzzing sequences.
35cargo fuzz run compile --features do_fuzz -- -seed_inputs=@fuzz/seed_inputs/compile/list.txt -runs=10 -max_total_time=5 -only_ascii=1
36```
37
38## Purifying the Bugs Found with Fuzzing
39
40<details><summary>The commands below were executed in a branch based on the following commit in "main" (click this line).</summary>
41
42```log
43commit e51a8b6f145be23fc2358b2cf0bab6707a7a46a0 (origin/main, origin/HEAD, main)
44Author: Bill Ticehurst <billti@microsoft.com>
45Date: Wed Apr 19 10:42:03 2023 -0700
46
47 Fix mapping of spans for non-ASCII code (#182)
48
49 This builds on the branch for the PR at
50 https://github.com/microsoft/qsharp/pull/180 (which fixes the code
51 sharing issue with non-ASCII chars), not not strictly dependent.
52
53 The excessive comments on the `mapUtf8UnitsToUtf16Units` function should
54 outline why this is needed and what it fixes.
55```
56
57</details>
58
59The fuzzing run `cargo fuzz run compile`, if hits a panic, reports the panic message
60
61```log
62thread '<unnamed>' panicked at 'local variable should have inferred type', \
63 .../qsharp/compiler/qsc_frontend/src/typeck/rules.rs:326:30
64```
65
66Among the last few lines the log lists the following commands of interest:
67
68```log
69Reproduce with:
70 cargo fuzz run compile fuzz/artifacts/compile/crash-22fc59256083904ead44f3ce8f5f04a251d7cc23
71Minimize test case with:
72 cargo fuzz tmin compile fuzz/artifacts/compile/crash-22fc59256083904ead44f3ce8f5f04a251d7cc23
73```
74
75The first thing you may typically do is to look at the input that caused the panic:
76`cat fuzz/artifacts/compile/crash-22fc59256083904ead44f3ce8f5f04a251d7cc23`.
77The input may be longer than sufficient to cause the panic. So the next thing you may want to do
78is to shorten the input (see the "Minimize test case with" above), but it is recommended to
79add `-r 10000` after `tmin` (which results in a longer run but much shorter input, in any case the run takes within one minute):
80`cargo fuzz tmin -r 10000 compile fuzz/artifacts/compile/crash-22fc59256083904ead44f3ce8f5f04a251d7cc23`.
81This command makes a number of runs with shorter input to figure out a shorter sequence that causes the panic.
82The log fragments of interest are in the end:
83
84```log
85Minimized artifact:
86 fuzz/artifacts/compile/minimized-from-b665e6267c297608e85c5948481cd353107a07fa
87```
88
89```log
90Reproduce with:
91 cargo fuzz run compile fuzz/artifacts/compile/minimized-from-b665e6267c297608e85c5948481cd353107a07fa
92```
93
94**NOTE:** This command of automated input shortening can end up in a _different panic_.
95That panic can be a new bug found or a previously known bug.
96
97Make sure that you are still on track, reproduce the panic of interest with the shortened input (see "Reproduce with" command above):
98
99```bash
100cargo fuzz run compile --features do_fuzz fuzz/artifacts/compile/minimized-from-b665e6267c297608e85c5948481cd353107a07fa
101```
102
103Right below the panic message the log gives you a stack trace hint:
104`note: run with 'RUST_BACKTRACE=1' environment variable to display a backtrace`.
105
106You can enable the stack trace display, when reproducing the panic, like this:
107
108```bash
109RUST_BACKTRACE=1 cargo fuzz run compile --features do_fuzz fuzz/artifacts/compile/minimized-from-b665e6267c297608e85c5948481cd353107a07fa
110```
111
112(you repeat the repro command but you set the environment variable `RUST_BACKTRACE` for that run only).
113
114**NOTE:** See the stack trace shown not in the end of the repro log, but immediately after the panic message.
115
116<details><summary>Example (click this line).</summary>
117
118```log
119thread 'unnamed' panicked at 'local variable should have inferred type', /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_frontend/src/typeck/rules.rs:326:30
120stack backtrace:
121 0: rust_begin_unwind
122 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/panicking.rs:577:5
123 1: core::panicking::panic_fmt
124 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/panicking.rs:67:14
125 2: core::panicking::panic_display
126 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/panicking.rs:150:5
127 3: core::panicking::panic_str
128 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/panicking.rs:134:5
129 4: core::option::expect_failed
130 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/option.rs:2025:5
131 5: core::option::Option{T}::expect
132 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/option.rs:913:21
133 6: qsc_frontend::typeck::rules::Context::infer_expr
134 at ./src/typeck/rules.rs:326:30
135 7: qsc_frontend::typeck::rules::Context::infer_binop
136 at ./src/typeck/rules.rs:445:32
137 8: qsc_frontend::typeck::rules::Context::infer_expr
138 at ./src/typeck/rules.rs:217:56
139 9: qsc_frontend::typeck::rules::Context::infer_binop
140 at ./src/typeck/rules.rs:444:32
141 10: qsc_frontend::typeck::rules::Context::infer_expr
142 at ./src/typeck/rules.rs:217:56
143 11: qsc_frontend::typeck::rules::Context::infer_update
144 at ./src/typeck/rules.rs:509:38
145 12: qsc_frontend::typeck::rules::Context::infer_expr
146 at ./src/typeck/rules.rs:368:27
147 13: qsc_frontend::typeck::rules::Context::infer_update
148 at ./src/typeck/rules.rs:509:38
149 14: qsc_frontend::typeck::rules::Context::infer_expr
150 at ./src/typeck/rules.rs:368:27
151 15: qsc_frontend::typeck::rules::Context::infer_update
152 at ./src/typeck/rules.rs:509:38
153 16: qsc_frontend::typeck::rules::Context::infer_expr
154 at ./src/typeck/rules.rs:368:27
155 17: qsc_frontend::typeck::rules::Context::infer_update
156 at ./src/typeck/rules.rs:509:38
157 18: qsc_frontend::typeck::rules::Context::infer_expr
158 at ./src/typeck/rules.rs:368:27
159 19: qsc_frontend::typeck::rules::Context::infer_update
160 at ./src/typeck/rules.rs:509:38
161 20: qsc_frontend::typeck::rules::Context::infer_expr
162 at ./src/typeck/rules.rs:368:27
163 21: qsc_frontend::typeck::rules::Context::infer_update
164 at ./src/typeck/rules.rs:509:38
165 22: qsc_frontend::typeck::rules::Context::infer_expr
166 at ./src/typeck/rules.rs:368:27
167 23: qsc_frontend::typeck::rules::Context::infer_update
168 at ./src/typeck/rules.rs:509:38
169 24: qsc_frontend::typeck::rules::Context::infer_expr
170 at ./src/typeck/rules.rs:368:27
171 25: qsc_frontend::typeck::rules::Context::infer_update
172 at ./src/typeck/rules.rs:509:38
173 26: qsc_frontend::typeck::rules::Context::infer_expr
174 at ./src/typeck/rules.rs:368:27
175 27: qsc_frontend::typeck::rules::Context::infer_expr
176 at ./src/typeck/rules.rs:351:26
177 28: qsc_frontend::typeck::rules::Context::infer_stmt
178 at ./src/typeck/rules.rs:172:27
179 29: qsc_frontend::typeck::rules::Context::infer_block
180 at ./src/typeck/rules.rs:143:35
181 30: qsc_frontend::typeck::rules::Context::infer_spec
182 at ./src/typeck/rules.rs:106:21
183 31: qsc_frontend::typeck::rules::spec
184 at ./src/typeck/rules.rs:610:5
185 32: qsc_frontend::typeck::check::Checker::check_spec
186 at ./src/typeck/check.rs:105:22
187 33: {qsc_frontend::typeck::check::Checker as qsc_ast::visit::Visitor}::visit_callable_decl
188 at ./src/typeck/check.rs:131:48
189 34: qsc_ast::visit::walk_item
190 at /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_ast/src/visit.rs:94:37
191 35: qsc_ast::visit::Visitor::visit_item
192 at /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_ast/src/visit.rs:20:9
193 36: qsc_ast::visit::walk_namespace::{{closure}}
194 at /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_ast/src/visit.rs:86:41
195 37: {core::slice::iter::Iter{T} as core::iter::traits::iterator::Iterator}::for_each
196 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/core/src/slice/iter/macros.rs:201:21
197 38: qsc_ast::visit::walk_namespace
198 at /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_ast/src/visit.rs:86:5
199 39: qsc_ast::visit::Visitor::visit_namespace
200 at /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp/compiler/qsc_ast/src/visit.rs:16:9
201 40: {qsc_frontend::typeck::check::Checker as qsc_ast::visit::Visitor}::visit_package
202 at ./src/typeck/check.rs:118:13
203 41: qsc_frontend::compile::typeck_all
204 at ./src/compile.rs:318:5
205 42: qsc_frontend::compile::compile
206 at ./src/compile.rs:175:28
207 43: compile::_::__libfuzzer_sys_run
208 at ./fuzz/fuzz_targets/compile.rs:10:17
209 44: rust_fuzzer_test_input
210 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/src/lib.rs:224:17
211 45: libfuzzer_sys::test_input_wrap::{{closure}}
212 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/src/lib.rs:61:9
213 46: std::panicking::try::do_call
214 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/panicking.rs:485:40
215 47: __rust_try
216 48: std::panicking::try
217 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/panicking.rs:449:19
218 49: std::panic::catch_unwind
219 at /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/panic.rs:140:14
220 50: LLVMFuzzerTestOneInput
221 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/src/lib.rs:59:22
222 51: _ZN6fuzzer6Fuzzer15ExecuteCallbackEPKhm
223 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/libfuzzer/FuzzerLoop.cpp:612:13
224 52: _ZN6fuzzer10RunOneTestEPNS_6FuzzerEPKcm
225 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/libfuzzer/FuzzerDriver.cpp:324:6
226 53: _ZN6fuzzer12FuzzerDriverEPiPPPcPFiPKhmE
227 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/libfuzzer/FuzzerDriver.cpp:860:9
228 54: main
229 at /home/rokuzmin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.6/libfuzzer/FuzzerMain.cpp:20:10
230 55: __libc_start_main
231 at /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
232 56: _start
233note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
234==16693== ERROR: libFuzzer: deadly signal
235```
236
237</details>
238
239After the steps above you typically get all the necessary information to file (or to work on) a bug - the short enough input and the panic stack trace.
240
241If the input is still too long then you may want to shorten it manually (e.g. remove the Q# code comments from the Q# input).
242
243If you believe that the input is still longer than sufficient to reproduce the panic, e.g. the panic complains about a local variable in the Q# input,
244and in the Q# input you have a dozen of functions with a few dozens of nested scopes with local variables, then you will likely want to break in the debugger
245upon panic and see the particular local variable that caused the panic.
246
247To achieve that, you need to rebuild the fuzzing binary with the debugging information (`--dev`):
248`cargo fuzz build --dev compile`.
249The resulting binary "compile" should be in the "debug", not "release", directory
250
251```bash
252ls fuzz/target/x86_64-unknown-linux-gnu/debug/
253# ... compile ...
254```
255
256In your WSL session go to the root directory ("qsharp") of this repo and launch VSCode
257
258```bash
259code .
260```
261
262(Assuming your VSCode has CodeLLDB extension installed)
263
264- Click the "Run and Debug" view on the left (or press `<Ctrl+Shift+d>`).
265- Click the "create a launch.json file" link. Select debugger "LLDB".
266- Feel free to reply "No" to the question
267 "Cargo.toml has been detected in the workspace.
268 Would you like to generate launch configurations for its targets?".
269
270<details><summary>Change the contents of "launch.json" to look like this (click this line).</summary>
271
272```json
273{
274 // Use IntelliSense to learn about possible attributes.
275 // Hover to view descriptions of existing attributes.
276 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
277 "version": "0.2.0",
278 "configurations": [
279 {
280 "type": "lldb",
281 "request": "launch",
282 "name": "Debug",
283 "program": "${workspaceFolder}/fuzz/target/x86_64-unknown-linux-gnu/debug/compile",
284 "args": [
285 "fuzz/artifacts/compile/minimized-from-b665e6267c297608e85c5948481cd353107a07fa"
286 ],
287 "cwd": "${workspaceFolder}"
288 }
289 ]
290}
291```
292
293</details>
294
295- Press `<F5>` to run the debugging session. The debugger will stop upon panic.
296- Look at the call stack, click the stack frames of interest and inspect the local variables and
297 parameters in those frames to figure out the exact input fragment that caused the panic.
298
299Then you can manually minimize the input around the fragment of interest.
300
301## Adding More Fuzzing Targets
302
303```bash
304cargo fuzz add <new_fuzzing_target_identifier>
305cargo fuzz list # Optional. See the available fuzzing targets.
306# Edit the "fuzz/fuzz_targets/<new_fuzzing_target_identifier>.rs".
307cargo fuzz build # Optional. Build the fuzzing targets.
308cargo fuzz run <new_fuzzing_target_identifier> # Build and run.
309# See "Running" section for fine-tuning the runs.
310```
311
312## Adding More Seed Inputs for Fuzzing
313
314Add more files with input sequences to the
315"fuzz/seed_inputs/\<fuzzing_target>/" directory and add their paths to the list in
316"fuzz/seed_inputs/\<fuzzing_target>/list.txt".
317
318Details
319
320```bash
321cargo fuzz run compile --features do_fuzz -- -help=1 2>&1 | grep seed_inputs
322# seed_inputs 0 A comma-separated list of input files to use as an additional seed corpus.
323# Alternatively, an "@" followed by the name of a file containing the comma-separated list.
324```
325
326See more in [LibFuzzer Corpus](https://llvm.org/docs/LibFuzzer.html#corpus).
327
328## Code Coverage During Fuzzing
329
330Based on [Code Coverage](https://rust-fuzz.github.io/book/cargo-fuzz/coverage.html#code-coverage).
331
332Tested in WSL Ubuntu 22.04.
333
334### Code Coverage Prerequisites
335
336Note: The command `sudo apt install clang` installed `clang-10` and created the executables `clang` and `clang++` available in the `PATH`.
337The installation of other versions, like `sudo apt install clang-14` was installing the executables `clang-14` and `clang++-14`,
338but the executables `clang` and `clang++` were still of version 10.
339
340For the subsequent steps to succeed the executables `llvm-profdata` and `llvm-cov` need to be in the `PATH`.
341
342```bash
343which llvm-profdata<tab><tab> # See if the `llvm-profdata` is available.
344# llvm-profdata-10 # Not available, but version 10 is installed.
345which llvm-profdata-10 # See the path of version 10.
346# /usr/bin/llvm-profdata-10 # The path of version 10.
347pushd /usr/bin # Temporarily enter the dir where `llvm-profdata-10` is located.
348sudo ln -s llvm-profdata-10 llvm-profdata # Create symlink `llvm-profdata` -> `llvm-profdata-10`.
349sudo ln -s llvm-cov-10 llvm-cov # Create symlink `llvm-cov` -> `llvm-cov-10`.
350popd # Get back to the original dir.
351
352llvm-cov --version # Optional. See the version.
353#LLVM (http://llvm.org/):
354# LLVM version 10.0.0
355# Optimized build.
356# Default target: x86_64-pc-linux-gnu
357# Host CPU: skylake
358```
359
360The executables `llvm-profdata` and `llvm-cov` also need to be in the nightly toolchain.
361
362```bash
363rustup default # Make sure that the nightly toolchain is the default.
364# nightly-x86_64-unknown-linux-gnu (default)
365
366# See if the executables `llvm-profdata` and `llvm-cov` are installed in the nightly toolchain:
367ls /home/$USER/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
368# gcc-ld llc # Nothing starting with `llvm-`.
369
370# Not installed, install:
371rustup component add --toolchain nightly llvm-tools-preview
372
373# Make sure they are installed:
374ls /home/$USER/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
375# gcc-ld llc llvm-ar llvm-as llvm-cov llvm-dis llvm-nm llvm-objcopy llvm-objdump llvm-profdata llvm-readobj llvm-size llvm-strip opt rust-lld
376
377# Optional. See the version:
378/home/$USER/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-cov --version
379#LLVM (http://llvm.org/):
380# LLVM version 16.0.2-rust-1.70.0-nightly
381# Optimized build.
382```
383
384Install the Rust demangler.
385`cargo install rustfilt`
386
387### Running the Code Coverage Tool
388
389```bash
390# In "qsc_frontend" directory:
391
392# Make sure that fuzzing still works OK:
393cargo fuzz list # Optional. See the fuzzing targets.
394#compile
395cargo fuzz run compile --features do_fuzz -- -seed_inputs=@fuzz/seed_inputs/compile/list.txt -max_total_time=1
396 # Run the fuzzing for at least 1 second.
397 # It is assumed that earlier you were running `cargo fuzz run compile` for a long time to gather
398 # the execution statistics.
399
400cargo fuzz coverage compile # Gather the code coverage info.
401 # The run takes a few minutes.
402# Later you will likely need the following data:
403# One of the first log lines shows the absolute path to the executable the code coverage is gathered for:
404# .../target/x86_64-unknown-linux-gnu/coverage/x86_64-unknown-linux-gnu/release/compile
405# The last log line shows the absolute path to the file containing the code coverage info:
406# .../fuzz/coverage/compile/coverage.profdata
407
408# Generate the HTML-report showing the code coverage for the fuzzing executable:
409/home/$USER/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-cov \
410 show -Xdemangler=rustfilt -show-line-counts-or-regions -show-instantiations --ignore-filename-regex="/home/$USER/.cargo/.*" \
411 -format=html \
412 -instr-profile=fuzz/coverage/compile/coverage.profdata \
413 target/x86_64-unknown-linux-gnu/coverage/x86_64-unknown-linux-gnu/release/compile \
414 > index.html
415# The unrelated error and warning that were observed (did not affect the result):
416#error: /rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/sys/common/thread_local/fast_local.rs: No such file or directory
417#warning: The file '/rustc/88fb1b922b047981fc0cfc62aa1418b4361ae72e/library/std/src/sys/common/thread_local/fast_local.rs' isn't covered.
418
419# Open the "index.html" in the web-browser to see the code coverage report (not necessarily in WSL).
420```
421