microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/tests/vmm.md
254lines · modecode
| 1 | # VMM Tests |
| 2 | |
| 3 | The OpenVMM repo contains a set of "heavyweight" VMM tests that fully boot a |
| 4 | virtual machine and run validation against it. Unlike Unit tests, these are all |
| 5 | centralized in a single top-level `vmm_tests` directory. |
| 6 | |
| 7 | The OpenVMM PR and CI pipelines will run the full test suite on all supported |
| 8 | platforms; you'd typically run only the tests relevant to the changes you're |
| 9 | working on. |
| 10 | |
| 11 | ## Writing VMM Tests |
| 12 | |
| 13 | To streamline the process of booting and interacting with VMs during VMM tests, the |
| 14 | OpenVMM project uses an in-house test framework/library called `petri`. |
| 15 | |
| 16 | The library does not yet have a stable API, so at this time, the best way to |
| 17 | learn how to write new VMM tests is by reading through the existing corpus of |
| 18 | tests (start with vmm_tests/vmm_tests/tests/tests/multiarch.rs), |
| 19 | as well as reading through `petri`'s rustdoc-generated API docs. |
| 20 | |
| 21 | The tests are currently generated using a macro (`#[vmm_test]`) that allows |
| 22 | the same test body to be run in a variety of scenarios, with different guest |
| 23 | operating systems, firmwares, and VMMs (including Hyper-V, which is useful |
| 24 | for testing certain OpenHCL features that aren't supported when using |
| 25 | OpenVMM as the host VMM). |
| 26 | |
| 27 | ### "heavy" tests |
| 28 | |
| 29 | The global [nextest.toml](https://github.com/microsoft/openvmm/blob/main/.config/nextest.toml) |
| 30 | configures how tests run in our test environments. The `default` and `ci` profiles |
| 31 | control things like timeouts, and how many resources we allocate to a given test. The number |
| 32 | of required threads is a fuzzy requirement relative to the number of VPs consumed by the VM under |
| 33 | test, the amount of memory your test needs, the host test framework (petri itself), and so on. |
| 34 | |
| 35 | We have some pre-defined overrides that perform filterset matching on test name. These overrides |
| 36 | are curated to balance individual trial (test case) performance against overall concurrency on |
| 37 | engineers' local machines and in CI. Put these special words in your test to opt in to that override: |
| 38 | |
| 39 | - `heavy` - if your test is heavier than the typical vmm_test. E.g., your test explicitly requests 16 virtual processors. |
| 40 | - `very_heavy` if your test is heavier than a `heavy` test. E.g., your test explicitly requests 32 virtual processors. |
| 41 | |
| 42 | ### "unstable" tests |
| 43 | |
| 44 | If a test is not yet reliable enough to gate PRs, add `unstable` to the macro. |
| 45 | |
| 46 | For individual variants: |
| 47 | |
| 48 | ```rust,ignore |
| 49 | #[vmm_test( |
| 50 | // unstable variant: |
| 51 | unstable_hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)), |
| 52 | // other reliable variants: |
| 53 | hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) |
| 54 | // ... |
| 55 | )] |
| 56 | async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> { |
| 57 | // ... |
| 58 | } |
| 59 | ``` |
| 60 | |
| 61 | For all variants of the test: |
| 62 | |
| 63 | ```rust,ignore |
| 64 | #[vmm_test_with(unstable( |
| 65 | hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)), |
| 66 | hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) |
| 67 | // ... |
| 68 | ))] |
| 69 | async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> { |
| 70 | // ... |
| 71 | } |
| 72 | ``` |
| 73 | |
| 74 | Unstable tests run in the same CI job as stable tests. When an unstable test fails |
| 75 | the CI run will pass with a warning |
| 76 | |
| 77 | To promote an unstable test to stable, remove `unstable` from the macro. This is |
| 78 | a single-place change — no CI or configuration updates are required. |
| 79 | |
| 80 | To ignore these `unstable` tags and report failures for all tests when running |
| 81 | locally, set the following environment variable: `PETRI_REPORT_UNSTABLE_FAIL=1` |
| 82 | |
| 83 | ## Running VMM Tests (Flowey) |
| 84 | |
| 85 | The easiest way to run VMM tests locally is `cargo xflowey vmm-tests-run`. It |
| 86 | automatically discovers required artifacts, builds dependencies, and runs your |
| 87 | tests in a single command. |
| 88 | |
| 89 | To run a **specific test** (or set of tests), use `--filter` with a |
| 90 | [nextest filter](https://nexte.st/docs/filtersets/) expression: |
| 91 | |
| 92 | ```bash |
| 93 | cargo xflowey vmm-tests-run --filter "test(my_test_name)" --dir /tmp/vmm-tests-run |
| 94 | ``` |
| 95 | |
| 96 | ### Targeting a Platform |
| 97 | |
| 98 | By default, `vmm-tests-run` builds for the current host. Use `--target` to |
| 99 | build for a different platform. The supported targets are: |
| 100 | |
| 101 | | Target | Description | |
| 102 | |--------|-------------| |
| 103 | | `windows-x64` | Windows x86_64 (Hyper-V / WHP) | |
| 104 | | `windows-aarch64` | Windows ARM64 (Hyper-V / WHP) | |
| 105 | | `linux-x64` | Linux x86_64 | |
| 106 | |
| 107 | **Cross-compiling for Windows from WSL2** is fully supported, you can build |
| 108 | and run Windows VMM tests directly from your WSL2 shell. This requires the |
| 109 | cross-compilation environment to be set up first. |
| 110 | |
| 111 | Then target Windows as usual. The output directory **must** be on the Windows |
| 112 | filesystem (e.g., `/mnt/d/...`): |
| 113 | |
| 114 | ```bash |
| 115 | cargo xflowey vmm-tests-run --target windows-x64 --dir /mnt/d/vmm_tests |
| 116 | ``` |
| 117 | |
| 118 | For full cross-compilation setup instructions, see |
| 119 | [Cross Compiling for Windows](../getting_started/cross_compile.md). |
| 120 | |
| 121 | When running Hyper-V tests, your user account must be a member of the |
| 122 | Hyper-V Administrators group. |
| 123 | |
| 124 | To see all available options: `cargo xflowey vmm-tests-run --help`. |
| 125 | |
| 126 | ### Lazy Fetch (Default) |
| 127 | |
| 128 | By default, VHD/ISO disk images are streamed on demand via HTTP and cached |
| 129 | locally in a SQLite database. This avoids multi-GB upfront downloads and |
| 130 | significantly speeds up the dev inner loop once the cache is warm. |
| 131 | |
| 132 | Lazy fetch is automatically disabled for Hyper-V tests, which require local |
| 133 | files. To force all images to be downloaded upfront, pass `--no-lazy-fetch`: |
| 134 | |
| 135 | ```bash |
| 136 | cargo xflowey vmm-tests-run --filter "test(my_test)" --dir /tmp/vmm-tests-run --no-lazy-fetch |
| 137 | ``` |
| 138 | |
| 139 | ## Running VMM Tests (Manual) |
| 140 | |
| 141 | ```admonish tip |
| 142 | Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM |
| 143 | tests. It is a significant improvement over the built-in `cargo test` runner, |
| 144 | and is the test runner we use in all our CI pipelines. |
| 145 | |
| 146 | You can install it locally by running: `cargo install cargo-nextest --locked` |
| 147 | |
| 148 | See the [cargo-nextest](https://nexte.st/) documentation for more info. |
| 149 | ``` |
| 150 | |
| 151 | You can directly invoke `cargo test` or `cargo nextest` to run the vmm |
| 152 | tests manually. |
| 153 | |
| 154 | Unlike Unit Tests, VMM tests may rely on additional external artifacts in order |
| 155 | to run. e.g: Virtual Disk Images, pre-built OpenHCL binaries, UEFI / PCAT |
| 156 | firmware blobs, etc. |
| 157 | |
| 158 | As such, the first step in running a VMM test is to ensure you have acquired all |
| 159 | external test artifacts it may depend upon. |
| 160 | |
| 161 | The VMM test infrastructure does not automatically fetch / rebuild |
| 162 | necessary artifacts unless you are using [flowey](#running-vmm-tests-flowey). |
| 163 | However, the test infrastructure is designed to report clear |
| 164 | and actionable error messages whenever a required test artifact cannot be found, |
| 165 | which provide detailed instructions on how to build / acquire the missing |
| 166 | artifact. Some dependencies can only be built on Linux (OpenHCL and Linux |
| 167 | pipette, for example). If you are building on Linux and want to run Windows |
| 168 | guest tests, pipette will need to be |
| 169 | [cross compiled for Windows](#linux-cross-compiling-pipetteexe). |
| 170 | |
| 171 | ```admonish warning |
| 172 | `cargo nextest run` won't rebuild any of your changes. Make sure you `cargo build` |
| 173 | or `cargo xflowey igvm [RECIPE]` first! |
| 174 | ``` |
| 175 | |
| 176 | VMM tests are run using standard Rust test infrastructure, and are invoked via |
| 177 | `cargo test` / `cargo nextest`. |
| 178 | |
| 179 | ```bash |
| 180 | cargo nextest run -p vmm_tests [TEST_FILTERS] |
| 181 | ``` |
| 182 | |
| 183 | For example, to run a simple VMM test that simply boots using UEFI: |
| 184 | |
| 185 | ```bash |
| 186 | cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_frontpage |
| 187 | ``` |
| 188 | |
| 189 | And, for further example, to rebuild everything* and run all* the tests |
| 190 | (see below for details on these steps): |
| 191 | |
| 192 | *This will not work for Hyper-V tests. TMK tests need additional build steps. |
| 193 | |
| 194 | ```bash |
| 195 | # Install (most) of the dependencies; cargo nextest run may tell you |
| 196 | # about other deps. |
| 197 | rustup target add x86_64-unknown-none |
| 198 | rustup target add x86_64-unknown-uefi |
| 199 | rustup target add x86_64-pc-windows-msvc |
| 200 | sudo apt install clang-tools-14 lld-14 |
| 201 | |
| 202 | cargo install cargo-nextest --locked |
| 203 | |
| 204 | cargo xtask guest-test download-image |
| 205 | cargo xtask guest-test uefi --bootx64 |
| 206 | |
| 207 | # Rebuild all, and run all tests |
| 208 | cargo build --target x86_64-pc-windows-msvc -p pipette |
| 209 | cargo build --target x86_64-unknown-linux-musl -p pipette |
| 210 | |
| 211 | cargo build --target x86_64-pc-windows-msvc -p openvmm |
| 212 | |
| 213 | cargo xflowey build-igvm x64-test-linux-direct |
| 214 | cargo xflowey build-igvm x64-cvm |
| 215 | cargo xflowey build-igvm x64 |
| 216 | |
| 217 | cargo nextest run --target x86_64-pc-windows-msvc -p vmm_tests --filter-expr 'all() & !test(hyperv) & !test(tmk)' |
| 218 | ``` |
| 219 | |
| 220 | ### \[Linux] Cross-compiling `pipette.exe` |
| 221 | |
| 222 | These commands might use the test agent (`pipette`) that is put inside the VM, |
| 223 | and if the host machine OS and the guest machine OS are different, a setup |
| 224 | is required for cross-building. The recommended approach is to use WSL2 and |
| 225 | cross-compile using the freely available Microsoft Visual Studio Build Tools |
| 226 | or Microsoft Visual Studio Community Edition as described in |
| 227 | [\[WSL2\] Cross Compiling from WSL2 to Windows](../getting_started/cross_compile.md) |
| 228 | |
| 229 | If that is not possible, here is another option that relies on [MinGW-w64](https://www.mingw-w64.org/) |
| 230 | and doesn't require installing Windows: |
| 231 | |
| 232 | ```bash |
| 233 | # Do 1 once, do 2 as needed. |
| 234 | # |
| 235 | # 1. Setup the toolchain |
| 236 | rustup target add x86_64-pc-windows-gnu |
| 237 | sudo apt-get install mingw-w64-x86-64-dev |
| 238 | mingw-genlib -a x86_64 ./support/pal/api-ms-win-security-base-private-l1-1-1.def |
| 239 | sudo mv libapi-ms-win-security-base-private-l1-1-1.a /usr/x86_64-w64-mingw32/lib |
| 240 | |
| 241 | # 2. Build Pipette (builds target/x86_64-pc-windows-gnu/debug/pipette.exe first) |
| 242 | cargo build --target x86_64-pc-windows-gnu -p pipette |
| 243 | ``` |
| 244 | |
| 245 | ```bash |
| 246 | # Run a test |
| 247 | cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot |
| 248 | ``` |
| 249 | |
| 250 | ### Printing logs for VMM Tests |
| 251 | |
| 252 | In order to see the OpenVMM logs while running a VMM test, do the following: |
| 253 | 1. Add the `--no-capture` flag to your `cargo nextest` command. |
| 254 | 2. Set `OPENVMM_LOG=trace`, replacing `trace` with the log level you want to view. |