microsoft/openvmm
Publicmirrored fromhttps://github.com/microsoft/openvmmAvailable
Guide/src/reference/openhcl/debugging.md
110lines · modecode
| 1 | # Debugging OpenHCL |
| 2 | |
| 3 | OpenHCL provides several debugging tools for investigating issues at the |
| 4 | user-mode and kernel level. See [ohcldiag-dev](./diag/ohcldiag_dev.md) for the |
| 5 | diagnostic client and [Tracing](./diag/tracing.md) for serial and event log |
| 6 | tracing. |
| 7 | |
| 8 | ## On-demand memory dumps |
| 9 | |
| 10 | Use `ohcldiag-dev dump` to capture a live user-mode memory dump of the OpenHCL |
| 11 | process at any time. The dump is an ELF core file that you can analyze with |
| 12 | `lldb`, `gdb`, or `rust-lldb`. See the |
| 13 | [ohcldiag-dev](./diag/ohcldiag_dev.md) page for the full command reference. |
| 14 | |
| 15 | ## User-mode crash dumps |
| 16 | |
| 17 | When an OpenHCL user-mode process crashes, a crash dump is automatically |
| 18 | generated via the `underhill-crash` infrastructure and sent to the host over |
| 19 | VMBus. On Windows hosts, these dumps are collected by Windows Error Reporting |
| 20 | (WER). Use `lldb` or `gdb` to analyze the resulting ELF core dump. |
| 21 | |
| 22 | ## Kernel crash dumps |
| 23 | |
| 24 | Kernel-mode crash dumps (kdump) are **not currently supported** in OpenHCL. The |
| 25 | OpenHCL kernel does not have `CONFIG_KDUMP` or `CONFIG_KEXEC` compiled in. If |
| 26 | the kernel panics, no dump is generated. The only diagnostic output is COM3 |
| 27 | serial (if enabled), which captures the panic message in real time. If the |
| 28 | diagnostic service was running before the panic, `ohcldiag-dev` may have ring |
| 29 | buffer messages up to that point, but it cannot capture the panic itself since |
| 30 | the service is terminated by the panic. |
| 31 | |
| 32 | For debugging kernel-level issues, the best approach is to enable serial output |
| 33 | via COM3 (see below) — it captures output from the very first instruction of |
| 34 | kernel boot. |
| 35 | |
| 36 | ## Getting OpenHCL kernel logs (COM3 vs ohcldiag-dev) |
| 37 | |
| 38 | Two methods exist for capturing OpenHCL kernel (`kmsg`) output: |
| 39 | |
| 40 | **COM3 serial** uses direct UART I/O — it streams output from the very first |
| 41 | instruction of OpenHCL boot in real time. |
| 42 | |
| 43 | **ohcldiag-dev** connects over vsock to the diagnostic service, which reads |
| 44 | `/dev/kmsg`. Because `/dev/kmsg` preserves the kernel ring buffer, early boot |
| 45 | messages are **replayed** when you connect — you get them even if you connect |
| 46 | late. However, `ohcldiag-dev` only works if the diagnostic service successfully |
| 47 | starts. |
| 48 | |
| 49 | | Boot phase | COM3 serial | ohcldiag-dev | |
| 50 | |------------|:-----------:|:------------:| |
| 51 | | Very early kernel (entry → memory setup) | ✅ live | ✅ replayed from ring buffer | |
| 52 | | Device initialization (VMBus, etc.) | ✅ live | ✅ replayed from ring buffer | |
| 53 | | Kernel panic before userspace | ✅ live | ❌ service never starts | |
| 54 | | Boot hang (kernel stuck) | ✅ live | ❌ service never starts | |
| 55 | | After diagnostic service starts | ✅ live | ✅ live | |
| 56 | |
| 57 | For most development, `ohcldiag-dev` is sufficient — boot succeeds and you get |
| 58 | logs. COM3 is essential for debugging early boot failures, kernel panics, and |
| 59 | init crashes. |
| 60 | |
| 61 | ## Enabling COM3 on Hyper-V |
| 62 | |
| 63 | COM3 support requires a host OS build that includes the `EnableAdditionalComPorts` |
| 64 | code path. This was added in Windows 11 26H1 (build 28000+, Insider Canary channel). |
| 65 | It is **not available** on Windows 11 24H2, 25H2, or Windows Server 2025. |
| 66 | |
| 67 | To enable COM3 on a supported build: |
| 68 | |
| 69 | ```powershell |
| 70 | # Enable additional COM ports (requires reboot or VMMS restart) |
| 71 | reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization" /v EnableAdditionalComPorts /t REG_DWORD /d 1 /f |
| 72 | |
| 73 | # Attach COM3 to a named pipe for a VM |
| 74 | Set-VMComPort -VMName $VmName -Number 3 -Path "\\.\pipe\openhcl-com3" |
| 75 | |
| 76 | # Read the serial output |
| 77 | hvc serial -c -p 3 -r $VmName |
| 78 | ``` |
| 79 | |
| 80 | ```admonish note |
| 81 | The `flowey` test runner (`install_vmm_tests_deps`) sets this registry key |
| 82 | automatically when running VMM tests. If you run `cargo xflowey` to execute |
| 83 | tests, you'll be prompted to allow the registry change. |
| 84 | ``` |
| 85 | |
| 86 | ## Recommended host OS for OpenHCL development |
| 87 | |
| 88 | We recommend running a **Windows 11 26H1** Insider flight (Canary channel, |
| 89 | build 28000+) on your development machine, if it is available for your device. |
| 90 | This gives you COM3 support via the registry key above, plus access to the |
| 91 | latest Hyper-V features. This matches the OS used on the project's self-hosted |
| 92 | CI runners. |
| 93 | |
| 94 | Note that 26H1 Insider builds may not be available for all hardware — see |
| 95 | [What to know about Windows 11 version 26H1](https://techcommunity.microsoft.com/blog/windows-itpro-blog/what-to-know-about-windows-11-version-26h1/4491941) |
| 96 | for details and the |
| 97 | [Windows Insider Flight Hub](https://learn.microsoft.com/en-us/windows-insider/flight-hub/) |
| 98 | for availability. |
| 99 | |
| 100 | If you're on Windows 11 24H2/25H2 (builds 26100/26200), COM3 is not available |
| 101 | via the registry key. Use `ohcldiag-dev` for kernel logs instead. |
| 102 | |
| 103 | ## ARM64 limitation |
| 104 | |
| 105 | On Hyper-V, additional serial ports (COM3+) are **not supported on ARM64**. The |
| 106 | Hyper-V serial device for ARM64 does not support ports beyond COM1 and COM2. On |
| 107 | ARM64 hosts, use `ohcldiag-dev` for OpenHCL kernel logs. |
| 108 | |
| 109 | This limitation is Hyper-V-specific — when running OpenVMM directly (without |
| 110 | Hyper-V), ARM64 serial output works via PL011 UART. |
| 111 | |