microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/tests/fuzzing/running.md
165lines · modecode
| 1 | # Running Fuzzers Locally |
| 2 | |
| 3 | ## Installing Dependencies |
| 4 | |
| 5 | To begin fuzzing in OpenVMM, you'll need to install `cargo-fuzz` and a nightly |
| 6 | rust compiler. |
| 7 | |
| 8 | Installation should be as simple as: |
| 9 | |
| 10 | ```bash |
| 11 | rustup install nightly |
| 12 | cargo install cargo-fuzz |
| 13 | ``` |
| 14 | |
| 15 | > NOTE: `cargo-fuzz` requires a nightly toolchain as it compiles targets with |
| 16 | > [ASAN](https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170) to |
| 17 | > improve the likelihood of finding bugs and the reproducibility of testcases. |
| 18 | |
| 19 | ## Running |
| 20 | |
| 21 | While its entirely possible to run the various fuzzers in the OpenVMM repo using |
| 22 | `cargo fuzz` directly, the OpenVMM repo includes additional tooling to streamline |
| 23 | working with fuzzers at "OpenVMM scale": `cargo xtask fuzz` |
| 24 | |
| 25 | `cargo xtask fuzz` bridges the gap between `cargo fuzz`'s "crate-oriented" |
| 26 | tooling, and OpenVMM's "repo-oriented" tooling. |
| 27 | |
| 28 | e.g: instead of manually navigating to each individual `crate/fuzz` directory in |
| 29 | order to use `cargo fuzz`, with `cargo xtask fuzz`, you can list/run/build _any_ |
| 30 | fuzzer in the OpenVMM repo, regardless where it happens to be in the repo! |
| 31 | |
| 32 | Before you can run a fuzzer, you need to know its name. To see a list of all |
| 33 | fuzzers currently in the OpenVMM tree, you can run: |
| 34 | |
| 35 | ```bash |
| 36 | cargo xtask fuzz list |
| 37 | ``` |
| 38 | |
| 39 | The output will be a list of available "fuzz targets": |
| 40 | |
| 41 | ``` |
| 42 | $ cargo xtask fuzz list |
| 43 | fuzz_chipset_battery |
| 44 | fuzz_ide |
| 45 | fuzz_scsi_buffers |
| 46 | ``` |
| 47 | |
| 48 | Once you've got a fuzzer you're interested in running (e.g: `fuzz_ide`), |
| 49 | starting a fuzzing session is as easy as running: |
| 50 | |
| 51 | ```bash |
| 52 | cargo xtask fuzz run fuzz_ide |
| 53 | ``` |
| 54 | |
| 55 | And you're off! If you see a whole bunch of terminal spew, congrats, you're |
| 56 | fuzzing! |
| 57 | |
| 58 | When run locally using the above command, the fuzzer will run indefinitely until |
| 59 | a crash is discovered. |
| 60 | |
| 61 | If you need to tweak the runtime behavior of the command, all of libFuzzer's |
| 62 | [commandline options][cli-opts] are at your disposal. Alternatively you can |
| 63 | print the help of the fuzzer like so: |
| 64 | |
| 65 | ```bash |
| 66 | # NOTE: The "-- --" is required to differentiate between `xtask fuzz`'s |
| 67 | # extra-args, and `cargo fuzz`'s extra-args |
| 68 | cargo xtask fuzz run fuzz_ide -- -- -help=1 |
| 69 | ``` |
| 70 | |
| 71 | [cli-opts]: https://www.llvm.org/docs/LibFuzzer.html#options |
| 72 | [toolchain-overrides-url]: https://rust-lang.github.io/rustup/overrides.html |
| 73 | |
| 74 | ## Other Fuzzing Commands |
| 75 | |
| 76 | The `cargo xtask fuzz` CLI includes plenty of docs via `--help` text. Don't be |
| 77 | afraid to dig into all the tools available via `cargo xtask fuzz` by using |
| 78 | `--help` at both the top-level, and for more details regarding the various |
| 79 | subcommands. |
| 80 | |
| 81 | Note that most `cargo xtask fuzz` commands mirror those from `cargo fuzz`, so |
| 82 | for additional information on how certain commands work, check out the |
| 83 | [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html). |
| 84 | |
| 85 | ## Coverage |
| 86 | |
| 87 | The effectiveness of fuzzing can be measured with code coverage. |
| 88 | |
| 89 | Code coverage can be analyzed to determine which branches in the target were |
| 90 | exercised and which were missed by the fuzzer. This can be used to determine if |
| 91 | the fuzzer needs improvements or is doing an adequate job. |
| 92 | |
| 93 | Before you begin you'll need some additional dependencies to generate an html |
| 94 | report: |
| 95 | |
| 96 | ```bash |
| 97 | rustup +nightly component add llvm-tools |
| 98 | apt install lcov |
| 99 | ``` |
| 100 | |
| 101 | To generate a report with "sane defaults", you can simply run: |
| 102 | |
| 103 | ```bash |
| 104 | cargo xtask fuzz coverage fuzz_ide --with-html-report |
| 105 | ``` |
| 106 | |
| 107 | Simply navigate to the `html/report/dir/index.html`` on your machine and inspect the coverage! |
| 108 | |
| 109 |  |
| 110 | |
| 111 | > NOTE: `--with-html-report` offers a quick-and-easy way for an individual user |
| 112 | > generate a coverage report locally, but it may not be entirely appropriate for |
| 113 | > more "industrial scale" fuzzing pipelines. |
| 114 | |
| 115 | ### Manual Coverage Generation (Advanced) |
| 116 | |
| 117 | The basic way this is done is by running all the discovered input testcases |
| 118 | through the fuzzer and merging all the coverage events together (remember, the |
| 119 | fuzzers only save testcases which generate new coverage). Cargo-fuzz provides a |
| 120 | way to do this with the `coverage` subcommand. This step generates a |
| 121 | `coverage.profdata` file which can be turned into a human-readable HTML report: |
| 122 | |
| 123 | ```bash |
| 124 | # cargo xtask fuzz coverage <fuzzer name> |
| 125 | cargo xtask fuzz coverage fuzz_ide |
| 126 | # confirm coverage.profdata was created |
| 127 | ls -l coverage.profdata |
| 128 | ``` |
| 129 | |
| 130 | OR if you have a large number of inputs (5k+) the below will collect and merge |
| 131 | coverage significantly faster: |
| 132 | |
| 133 | ```bash |
| 134 | # rebuild the fuzzer with coverage instrumentation |
| 135 | RUSTFLAGS="-C instrument-coverage" cargo +nightly fuzz build |
| 136 | # set env var to rustup's llvm-preview tools |
| 137 | LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-profdata')) |
| 138 | # make an output directory for corups minimation |
| 139 | mkdir min_corp |
| 140 | # run the minimizer putting the raw cov data into coverage.profraw |
| 141 | LLVM_PROFILE_FILE="coverage.profraw" ./fuzz/targets/<target-path>/release/fuzz_ide min_corp <path to input corpus directory> -merge=1 |
| 142 | # merge the raw data into coverage.profdata |
| 143 | $LLVM_TOOLS_PATH/llvm-profdata merge -sparse coverage.profraw -o coverage.profdata |
| 144 | ``` |
| 145 | |
| 146 | Next find the location of the llvm-tools you installed with rustup |
| 147 | (NOTE: rustup is used to install the LLVM tools to ensure that rust's llvm |
| 148 | version and the tool version are in sync), and convert the coverage data into |
| 149 | a report: |
| 150 | |
| 151 | ```bash |
| 152 | # set env var to rustup's llvm-preview tools |
| 153 | LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-profdata')) |
| 154 | # covert the coverage data into an lcov format |
| 155 | $LLVM_TOOLS_PATH/llvm-cov export -instr-profile=coverage.profdata \ |
| 156 | -format=lcov \ |
| 157 | -object ./fuzz/targets/<target-triple>/coverage/<target-triple>/release/fuzz_ide \ |
| 158 | --ignore-filename-regex "rustc" > coverage.lcov |
| 159 | # summarize the coverage information |
| 160 | lcov --summary ./coverage.lcov |
| 161 | # make an output directory for the html report |
| 162 | mkdir -p lcov_html |
| 163 | # generate the html report |
| 164 | genhtml -o lcov_html --legend --highlight ./coverage.lcov |
| 165 | ``` |
| 166 | |