microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/tests/vmm.md
198lines · modecode
| 1 | # VMM Tests |
| 2 | |
| 3 | The OpenVMM repo contains a set of "heavyweight" VMM tests that fully boot a |
| 4 | virtual machine and run validation against it. Unlike Unit tests, these are all |
| 5 | centralized in a single top-level `vmm_tests` directory. |
| 6 | |
| 7 | The OpenVMM PR and CI pipelines will run the full test suite on all supported |
| 8 | platforms; you'd typically run only the tests relevant to the changes you're |
| 9 | working on. |
| 10 | |
| 11 | ## Writing VMM Tests |
| 12 | |
| 13 | To streamline the process of booting and interacting with VMs during VMM tests, the |
| 14 | OpenVMM project uses an in-house test framework/library called `petri`. |
| 15 | |
| 16 | The library does not yet have a stable API, so at this time, the best way to |
| 17 | learn how to write new VMM tests is by reading through the existing corpus of |
| 18 | tests (start with vmm_tests/vmm_tests/tests/tests/multiarch.rs), |
| 19 | as well as reading through `petri`'s rustdoc-generated API docs. |
| 20 | |
| 21 | The tests are currently generated using a macro (`#[vmm_test]`) that allows |
| 22 | the same test body to be run in a variety of scenarios, with different guest |
| 23 | operating systems, firmwares, and VMMs (including Hyper-V, which is useful |
| 24 | for testing certain OpenHCL features that aren't supported when using |
| 25 | OpenVMM as the host VMM). |
| 26 | |
| 27 | ### "heavy" tests |
| 28 | |
| 29 | The global [nextest.toml](https://github.com/microsoft/openvmm/blob/main/.config/nextest.toml) |
| 30 | configures how tests run in our test environments. The `default` and `ci` profiles |
| 31 | control things like timeouts, and how many resources we allocate to a given test. The number |
| 32 | of required threads is a fuzzy requirement relative to the number of VPs consumed by the VM under |
| 33 | test, the amount of memory your test needs, the host test framework (petri itself), and so on. |
| 34 | |
| 35 | We have some pre-defined overrides that perform filterset matching on test name. These overrides |
| 36 | are curated to balance individual trial (test case) performance against overall concurrency on |
| 37 | engineers' local machines and in CI. Put these special words in your test to opt in to that override: |
| 38 | |
| 39 | - `heavy` - if your test is heavier than the typical vmm_test. E.g., your test explicitly requests 16 virtual processors. |
| 40 | - `very_heavy` if your test is heavier than a `heavy` test. E.g., your test explicitly requests 32 virtual processors. |
| 41 | |
| 42 | ### "unstable" tests |
| 43 | |
| 44 | If a test is not yet reliable enough to gate PRs, add the `_unstable` suffix to |
| 45 | the test function name. For example: |
| 46 | |
| 47 | ```rust,ignore |
| 48 | async fn my_new_boot_test_unstable(config: PetriVmConfig) -> Result<(), anyhow::Error> { |
| 49 | // ... |
| 50 | } |
| 51 | ``` |
| 52 | |
| 53 | Unstable tests run in the same CI job as stable tests. When an unstable test fails |
| 54 | the CI pass will pass with a warning |
| 55 | |
| 56 | To promote an unstable test to stable, rename the function to remove the |
| 57 | `_unstable` suffix. This is a single-place change — no CI or configuration |
| 58 | updates are required. |
| 59 | |
| 60 | ## Running VMM Tests (Flowey) |
| 61 | |
| 62 | The easiest way to run the VMM tests locally is using the |
| 63 | `cargo xflowey vmm-tests` command. To see the most up-to-date options, run: |
| 64 | `cargo xflowey vmm-tests --help`. When running Hyper-V tests, you will need |
| 65 | to use an administrator terminal window (this works even if you are running |
| 66 | from WSL2). When running Windows tests, the output dir should be on the |
| 67 | Windows file system. For example, from WSL2: |
| 68 | |
| 69 | ```bash |
| 70 | cargo xflowey vmm-tests --target windows-x64 --dir /mnt/e/vmm_tests |
| 71 | ``` |
| 72 | |
| 73 | This command will build or download all the test dependencies and copy them |
| 74 | to a self-contained folder that can be copied to another system for testing. |
| 75 | The folder will contain scripts for installing dependencies |
| 76 | (install_deps.ps1 on Windows) and running the tests (run.ps1 on Windows). |
| 77 | You can either specify a list of flags to disable certain tests and avoid |
| 78 | building/downloading some dependencies, or you can specify a custom |
| 79 | [nextest filter](https://nexte.st/docs/filtersets/) and list of artifacts. |
| 80 | In this case, all possible dependencies will be obtained since deriving them |
| 81 | from a test filter is not yet supported. |
| 82 | |
| 83 | ## Running VMM Tests (Manual) |
| 84 | |
| 85 | ```admonish tip |
| 86 | Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM |
| 87 | tests. It is a significant improvement over the built-in `cargo test` runner, |
| 88 | and is the test runner we use in all our CI pipelines. |
| 89 | |
| 90 | You can install it locally by running: `cargo install cargo-nextest --locked` |
| 91 | |
| 92 | See the [cargo-nextest](https://nexte.st/) documentation for more info. |
| 93 | ``` |
| 94 | |
| 95 | You can directly invoke `cargo test` or `cargo nextest` to run the vmm |
| 96 | tests manually. |
| 97 | |
| 98 | Unlike Unit Tests, VMM tests may rely on additional external artifacts in order |
| 99 | to run. e.g: Virtual Disk Images, pre-built OpenHCL binaries, UEFI / PCAT |
| 100 | firmware blobs, etc. |
| 101 | |
| 102 | As such, the first step in running a VMM test is to ensure you have acquired all |
| 103 | external test artifacts it may depend upon. |
| 104 | |
| 105 | The VMM test infrastructure does not automatically fetch / rebuild |
| 106 | necessary artifacts unless you are using [flowey](#running-vmm-tests-flowey). |
| 107 | However, the test infrastructure is designed to report clear |
| 108 | and actionable error messages whenever a required test artifact cannot be found, |
| 109 | which provide detailed instructions on how to build / acquire the missing |
| 110 | artifact. Some dependencies can only be built on Linux (OpenHCL and Linux |
| 111 | pipette, for example). If you are building on Linux and want to run Windows |
| 112 | guest tests, pipette will need to be |
| 113 | [cross compiled for Windows](#linux-cross-compiling-pipetteexe). |
| 114 | |
| 115 | ```admonish warning |
| 116 | `cargo nextest run` won't rebuild any of your changes. Make sure you `cargo build` |
| 117 | or `cargo xflowey igvm [RECIPE]` first! |
| 118 | ``` |
| 119 | |
| 120 | VMM tests are run using standard Rust test infrastructure, and are invoked via |
| 121 | `cargo test` / `cargo nextest`. |
| 122 | |
| 123 | ```bash |
| 124 | cargo nextest run -p vmm_tests [TEST_FILTERS] |
| 125 | ``` |
| 126 | |
| 127 | For example, to run a simple VMM test that simply boots using UEFI: |
| 128 | |
| 129 | ```bash |
| 130 | cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_frontpage |
| 131 | ``` |
| 132 | |
| 133 | And, for further example, to rebuild everything* and run all* the tests |
| 134 | (see below for details on these steps): |
| 135 | |
| 136 | *This will not work for Hyper-V tests. TMK tests need additional build steps. |
| 137 | |
| 138 | ```bash |
| 139 | # Install (most) of the dependencies; cargo nextest run may tell you |
| 140 | # about other deps. |
| 141 | rustup target add x86_64-unknown-none |
| 142 | rustup target add x86_64-unknown-uefi |
| 143 | rustup target add x86_64-pc-windows-msvc |
| 144 | sudo apt install clang-tools-14 lld-14 |
| 145 | |
| 146 | cargo install cargo-nextest --locked |
| 147 | |
| 148 | cargo xtask guest-test download-image |
| 149 | cargo xtask guest-test uefi --bootx64 |
| 150 | |
| 151 | # Rebuild all, and run all tests |
| 152 | cargo build --target x86_64-pc-windows-msvc -p pipette |
| 153 | cargo build --target x86_64-unknown-linux-musl -p pipette |
| 154 | |
| 155 | cargo build --target x86_64-pc-windows-msvc -p openvmm |
| 156 | |
| 157 | cargo xflowey build-igvm x64-test-linux-direct |
| 158 | cargo xflowey build-igvm x64-cvm |
| 159 | cargo xflowey build-igvm x64 |
| 160 | |
| 161 | cargo nextest run --target x86_64-pc-windows-msvc -p vmm_tests --filter-expr 'all() & !test(hyperv) & !test(tmk)' |
| 162 | ``` |
| 163 | |
| 164 | ### \[Linux] Cross-compiling `pipette.exe` |
| 165 | |
| 166 | These commands might use the test agent (`pipette`) that is put inside the VM, |
| 167 | and if the host machine OS and the guest machine OS are different, a setup |
| 168 | is required for cross-building. The recommended approach is to use WSL2 and |
| 169 | cross-compile using the freely available Microsoft Visual Studio Build Tools |
| 170 | or Microsoft Visual Studio Community Edition as described in |
| 171 | [\[WSL2\] Cross Compiling from WSL2 to Windows](../getting_started/cross_compile.md) |
| 172 | |
| 173 | If that is not possible, here is another option that relies on [MinGW-w64](https://www.mingw-w64.org/) |
| 174 | and doesn't require installing Windows: |
| 175 | |
| 176 | ```bash |
| 177 | # Do 1 once, do 2 as needed. |
| 178 | # |
| 179 | # 1. Setup the toolchain |
| 180 | rustup target add x86_64-pc-windows-gnu |
| 181 | sudo apt-get install mingw-w64-x86-64-dev |
| 182 | mingw-genlib -a x86_64 ./support/pal/api-ms-win-security-base-private-l1-1-1.def |
| 183 | sudo mv libapi-ms-win-security-base-private-l1-1-1.a /usr/x86_64-w64-mingw32/lib |
| 184 | |
| 185 | # 2. Build Pipette (builds target/x86_64-pc-windows-gnu/debug/pipette.exe first) |
| 186 | cargo build --target x86_64-pc-windows-gnu -p pipette |
| 187 | ``` |
| 188 | |
| 189 | ```bash |
| 190 | # Run a test |
| 191 | cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot |
| 192 | ``` |
| 193 | |
| 194 | ### Printing logs for VMM Tests |
| 195 | |
| 196 | In order to see the OpenVMM logs while running a VMM test, do the following: |
| 197 | 1. Add the `--no-capture` flag to your `cargo nextest` command. |
| 198 | 2. Set `OPENVMM_LOG=trace`, replacing `trace` with the log level you want to view. |
| 199 | |