microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/tests/fuzzing/running.md
254lines · modecode
| 1 | # Running Fuzzers Locally |
| 2 | |
| 3 | ## Installing Dependencies |
| 4 | |
| 5 | To begin fuzzing in OpenVMM, you'll need to install `cargo-fuzz` and a nightly |
| 6 | rust compiler. |
| 7 | |
| 8 | Installation should be as simple as: |
| 9 | |
| 10 | ```bash |
| 11 | rustup install nightly |
| 12 | cargo install cargo-fuzz |
| 13 | ``` |
| 14 | |
| 15 | For debugging crashes, a debugger such as `lldb` is useful: |
| 16 | |
| 17 | ```bash |
| 18 | sudo apt-get install -y lldb |
| 19 | ``` |
| 20 | |
| 21 | For coverage reports, install `llvm-tools` and `lcov`: |
| 22 | |
| 23 | ```bash |
| 24 | rustup +nightly component add llvm-tools |
| 25 | sudo apt-get install -y lcov |
| 26 | ``` |
| 27 | |
| 28 | ```admonish info |
| 29 | `cargo-fuzz` requires a nightly toolchain as it compiles targets with |
| 30 | [ASAN](https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170) to |
| 31 | improve the likelihood of finding bugs and the reproducibility of testcases. |
| 32 | ``` |
| 33 | |
| 34 | ```admonish warning |
| 35 | Fuzzing only works on **Linux**. libfuzzer-sys doesn't support Windows. |
| 36 | |
| 37 | On **aarch64**, you must set `RUSTFLAGS="-Ctarget-feature=+lse,+neon"` before |
| 38 | any cargo-fuzz command (build, run, coverage), or builds will fail with |
| 39 | atomics errors. This is not needed on x86_64. |
| 40 | ``` |
| 41 | |
| 42 | ## Running |
| 43 | |
| 44 | While it's entirely possible to run the various fuzzers in the OpenVMM repo using |
| 45 | `cargo fuzz` directly, the OpenVMM repo includes additional tooling to streamline |
| 46 | working with fuzzers at "OpenVMM scale": `cargo xtask fuzz` |
| 47 | |
| 48 | `cargo xtask fuzz` bridges the gap between `cargo fuzz`'s "crate-oriented" |
| 49 | tooling, and OpenVMM's "repo-oriented" tooling. |
| 50 | |
| 51 | e.g: instead of manually navigating to each individual `crate/fuzz` directory in |
| 52 | order to use `cargo fuzz`, with `cargo xtask fuzz`, you can list/run/build _any_ |
| 53 | fuzzer in the OpenVMM repo, regardless where it happens to be in the repo! |
| 54 | |
| 55 | Before you can run a fuzzer, you need to know its name. To see a list of all |
| 56 | fuzzers currently in the OpenVMM tree, you can run: |
| 57 | |
| 58 | ```bash |
| 59 | cargo xtask fuzz list |
| 60 | ``` |
| 61 | |
| 62 | The output will be a list of available "fuzz targets": |
| 63 | |
| 64 | ```bash |
| 65 | $ cargo xtask fuzz list |
| 66 | fuzz_chipset_battery |
| 67 | fuzz_ide |
| 68 | fuzz_scsi_buffers |
| 69 | ``` |
| 70 | |
| 71 | Once you've got a fuzzer you're interested in running (e.g: `fuzz_ide`), |
| 72 | starting a fuzzing session is as easy as running: |
| 73 | |
| 74 | ```bash |
| 75 | cargo xtask fuzz run fuzz_ide |
| 76 | ``` |
| 77 | |
| 78 | And you're off! If you see a whole bunch of terminal spew, congrats, you're |
| 79 | fuzzing! |
| 80 | |
| 81 | When run locally using the above command, the fuzzer will run indefinitely until |
| 82 | a crash is discovered. |
| 83 | |
| 84 | ### Building without running |
| 85 | |
| 86 | To just build a fuzzer without starting it: |
| 87 | |
| 88 | ```bash |
| 89 | cargo +nightly xtask fuzz build fuzz_ide |
| 90 | ``` |
| 91 | |
| 92 | The binary lands at `target/<triple>/release/fuzz_ide`. |
| 93 | |
| 94 | ### Reproducing a crash |
| 95 | |
| 96 | When LibFuzzer finds a crash, it saves the input as an artifact. Reproduce it: |
| 97 | |
| 98 | ```bash |
| 99 | # Through xtask (sets XTASK_FUZZ_REPRO=1 automatically for tracing) |
| 100 | cargo +nightly xtask fuzz run fuzz_ide path/to/crash-artifact |
| 101 | |
| 102 | # Or run the binary directly (faster for iteration) |
| 103 | ./target/<triple>/release/fuzz_ide path/to/crash-artifact |
| 104 | |
| 105 | # With tracing enabled (verbose — shows device state at each poll) |
| 106 | XTASK_FUZZ_REPRO=1 ./target/<triple>/release/fuzz_ide path/to/crash-artifact |
| 107 | ``` |
| 108 | |
| 109 | ### Minimizing crash inputs |
| 110 | |
| 111 | ```bash |
| 112 | cargo +nightly xtask fuzz tmin fuzz_ide path/to/crash-artifact |
| 113 | ``` |
| 114 | |
| 115 | ### Corpus management |
| 116 | |
| 117 | Corpus files live in `<crate>/fuzz/corpus/<target>/`. |
| 118 | Crash artifacts land in `<crate>/fuzz/artifacts/<target>/`. |
| 119 | |
| 120 | To minimize the corpus (remove redundant inputs): |
| 121 | |
| 122 | ```bash |
| 123 | cargo +nightly xtask fuzz cmin fuzz_ide |
| 124 | ``` |
| 125 | |
| 126 | ### Parallel fuzzing |
| 127 | |
| 128 | Use `-fork=N` to run N fuzzer processes in parallel across CPUs: |
| 129 | |
| 130 | ```bash |
| 131 | cargo +nightly xtask fuzz run fuzz_ide -- -- -fork=20 -max_total_time=3600 -print_final_stats=1 |
| 132 | ``` |
| 133 | |
| 134 | To survive crashes/timeouts/OOMs during long campaigns: |
| 135 | |
| 136 | ```bash |
| 137 | cargo +nightly xtask fuzz run fuzz_ide -- -- \ |
| 138 | -fork=20 -max_total_time=21600 \ |
| 139 | -ignore_crashes=1 -ignore_timeouts=1 -ignore_ooms=1 \ |
| 140 | -print_final_stats=1 |
| 141 | ``` |
| 142 | |
| 143 | If you need to tweak the runtime behavior of the command, all of libFuzzer's |
| 144 | [commandline options][cli-opts] are at your disposal. Alternatively you can |
| 145 | print the help of the fuzzer like so: |
| 146 | |
| 147 | ```bash |
| 148 | # NOTE: The "-- --" is required to differentiate between `xtask fuzz`'s |
| 149 | # extra-args, and `cargo fuzz`'s extra-args |
| 150 | cargo xtask fuzz run fuzz_ide -- -- -help=1 |
| 151 | ``` |
| 152 | |
| 153 | [cli-opts]: https://www.llvm.org/docs/LibFuzzer.html#options |
| 154 | |
| 155 | ## Other Fuzzing Commands |
| 156 | |
| 157 | The `cargo xtask fuzz` CLI includes plenty of docs via `--help` text. Don't be |
| 158 | afraid to dig into all the tools available via `cargo xtask fuzz` by using |
| 159 | `--help` at both the top-level, and for more details regarding the various |
| 160 | subcommands. |
| 161 | |
| 162 | Note that most `cargo xtask fuzz` commands mirror those from `cargo fuzz`, so |
| 163 | for additional information on how certain commands work, check out the |
| 164 | [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html). |
| 165 | |
| 166 | ## Coverage |
| 167 | |
| 168 | The effectiveness of fuzzing can be measured with code coverage. |
| 169 | |
| 170 | Code coverage can be analyzed to determine which branches in the target were |
| 171 | exercised and which were missed by the fuzzer. This can be used to determine if |
| 172 | the fuzzer needs improvements or is doing an adequate job. |
| 173 | |
| 174 | Before you begin you'll need some additional dependencies to generate an html |
| 175 | report: |
| 176 | |
| 177 | ```bash |
| 178 | rustup +nightly component add llvm-tools |
| 179 | apt install lcov |
| 180 | ``` |
| 181 | |
| 182 | To generate a report with "sane defaults", you can simply run: |
| 183 | |
| 184 | ```bash |
| 185 | cargo +nightly xtask fuzz coverage fuzz_ide --with-html-report |
| 186 | ``` |
| 187 | |
| 188 | Simply navigate to the `html/report/dir/index.html` on your machine and inspect the coverage! |
| 189 | |
| 190 | ```admonish tip |
| 191 | The overall coverage percentage in the HTML report covers *all* compiled code |
| 192 | including dependencies — it's typically 4–6% and meaningless. Focus on coverage |
| 193 | of the **target crate itself** (e.g., `ide/src/lib.rs`, `storvsp/src/lib.rs`). |
| 194 | ``` |
| 195 | |
| 196 |  |
| 197 | |
| 198 | ```admonish note |
| 199 | `--with-html-report` offers a quick-and-easy way for an individual user |
| 200 | generate a coverage report locally, but it may not be entirely appropriate for |
| 201 | more "industrial scale" fuzzing pipelines. |
| 202 | ``` |
| 203 | |
| 204 | ### Manual Coverage Generation (Advanced) |
| 205 | |
| 206 | The basic way this is done is by running all the discovered input testcases |
| 207 | through the fuzzer and merging all the coverage events together (remember, the |
| 208 | fuzzers only save testcases which generate new coverage). Cargo-fuzz provides a |
| 209 | way to do this with the `coverage` subcommand. This step generates a |
| 210 | `coverage.profdata` file which can be turned into a human-readable HTML report: |
| 211 | |
| 212 | ```bash |
| 213 | # cargo xtask fuzz coverage <fuzzer name> |
| 214 | cargo xtask fuzz coverage fuzz_ide |
| 215 | # confirm coverage.profdata was created |
| 216 | ls -l coverage.profdata |
| 217 | ``` |
| 218 | |
| 219 | OR if you have a large number of inputs (5k+) the below will collect and merge |
| 220 | coverage significantly faster: |
| 221 | |
| 222 | ```bash |
| 223 | # rebuild the fuzzer with coverage instrumentation |
| 224 | RUSTFLAGS="-C instrument-coverage" cargo +nightly fuzz build |
| 225 | # set env var to rustup's llvm-preview tools |
| 226 | LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-profdata')) |
| 227 | # make an output directory for corups minimation |
| 228 | mkdir min_corp |
| 229 | # run the minimizer putting the raw cov data into coverage.profraw |
| 230 | LLVM_PROFILE_FILE="coverage.profraw" ./fuzz/targets/<target-path>/release/fuzz_ide min_corp <path to input corpus directory> -merge=1 |
| 231 | # merge the raw data into coverage.profdata |
| 232 | $LLVM_TOOLS_PATH/llvm-profdata merge -sparse coverage.profraw -o coverage.profdata |
| 233 | ``` |
| 234 | |
| 235 | Next find the location of the llvm-tools you installed with rustup |
| 236 | (NOTE: rustup is used to install the LLVM tools to ensure that rust's llvm |
| 237 | version and the tool version are in sync), and convert the coverage data into |
| 238 | a report: |
| 239 | |
| 240 | ```bash |
| 241 | # set env var to rustup's llvm-preview tools |
| 242 | LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-profdata')) |
| 243 | # covert the coverage data into an lcov format |
| 244 | $LLVM_TOOLS_PATH/llvm-cov export -instr-profile=coverage.profdata \ |
| 245 | -format=lcov \ |
| 246 | -object ./fuzz/targets/<target-triple>/coverage/<target-triple>/release/fuzz_ide \ |
| 247 | --ignore-filename-regex "rustc" > coverage.lcov |
| 248 | # summarize the coverage information |
| 249 | lcov --summary ./coverage.lcov |
| 250 | # make an output directory for the html report |
| 251 | mkdir -p lcov_html |
| 252 | # generate the html report |
| 253 | genhtml -o lcov_html --legend --highlight ./coverage.lcov |
| 254 | ``` |
| 255 | |