microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/reference/dev_feats/gdbstub.md
167lines · modecode
| 1 | # Hardware Debugging (gdbstub) |
| 2 | |
| 3 | Think EXDI from Hyper-V, except instead of using the EXDI interface, we use the |
| 4 | [GDB Remote Serial Protocol](https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html) |
| 5 | (via the [`gdbstub`](https://github.com/daniel5151/gdbstub/) Rust library). |
| 6 | |
| 7 | Hardware debugging has several benefits over using an in-guest / in-kernel debugger: |
| 8 | |
| 9 | - Debugging early-boot scenarios (before UEFI / Windows / Linux debuggers are set up) |
| 10 | - Debugging low-level ISRs |
| 11 | - Non-intrusive debugging = easier to repro certain bugs |
| 12 | - Debugging SNP/TDX/VBS Confidential VMs |
| 13 | |
| 14 | ## Enabling the Debugger |
| 15 | |
| 16 | ### OpenVMM |
| 17 | 1. Pass the `--gdb <port>` flag at startup to enable the |
| 18 | debug worker. e.g., `--gdb 9001` |
| 19 | |
| 20 | To pause the VM until the debugger has been attached, pass `--paused` at startup. |
| 21 | |
| 22 | ### OpenHCL |
| 23 | 1. Pass the `OPENHCL_GDBSTUB=1` `OPENHCL_GDBSTUB_PORT=<gdbstub port>` parameters to enable gdbstub. e.g., `Set-VmFirmwareParameters -Name UhVM -CommandLine OPENHCL_GDBSTUB=1 OPENHCL_GDBSTUB_PORT=5900`. |
| 24 | 2. To expose a TCP port, run `ohcldiag-dev.exe <name> vsock-tcp-relay --allow-remote --reconnect <gdbstub port> <tcp port>`. |
| 25 | |
| 26 | To pause VTL0 boot until desired, pass `OPENHCL_VTL0_STARTS_PAUSED=1` as a parameter. Then once the debugger is attached, you can start VTL0 with `ohcldiag-dev.exe <name> resume`. |
| 27 | |
| 28 | ## Connecting via GDB |
| 29 | |
| 30 | The quickest way to get connected to a OpenVMM VM is via `gdb` directly. |
| 31 | |
| 32 | Note that GDB does _not_ support debugging PDBs, so if you're trying to debug |
| 33 | Windows, you'll be limited to plain disassembly. See the [`Connecting via |
| 34 | WinDbg`](#connecting-via-windbg) section below if this is your use-case. |
| 35 | |
| 36 | On the flipside, if you're trying to debug ELF images with DWARF debug info |
| 37 | (e.g., a vmlinux binary), then you'll likely want to use `gdb` directly, as it |
| 38 | will support source-level debugging with symbols, whereas WinDbg will not. |
| 39 | |
| 40 | You can install `gdb` via your distro's package manager. e.g., on Ubuntu: |
| 41 | |
| 42 | ```bash |
| 43 | sudo apt install gdb |
| 44 | ``` |
| 45 | |
| 46 | Once `gdb` is installed, run it, and enter the following `gdb` command (swapping |
| 47 | `9001` for whatever port you specified at the CLI) |
| 48 | |
| 49 | ```text |
| 50 | target remote :9001 |
| 51 | ``` |
| 52 | |
| 53 | If all goes well, you should get output similar to this: |
| 54 | |
| 55 | ```text |
| 56 | (gdb) target remote :9001 |
| 57 | Remote debugging using :9001 |
| 58 | warning: No executable has been specified and target does not support |
| 59 | determining executable automatically. Try using the "file" command. |
| 60 | 0xfffff8015c054c1f in ?? () |
| 61 | (gdb) |
| 62 | ``` |
| 63 | |
| 64 | At this point, you can try some basic GDB commands to make sure things are working. |
| 65 | |
| 66 | e.g., start / interrupt the VM's execution using `cont` and `ctrl-c` |
| 67 | |
| 68 | ```text |
| 69 | (gdb) cont |
| 70 | Continuing. |
| 71 | ^C # <-- hit ctrl-c in the terminal |
| 72 | Thread 1 received signal SIGINT, Interrupt. |
| 73 | 0xfffff8015c054c1f in ?? () |
| 74 | (gdb) |
| 75 | ``` |
| 76 | |
| 77 | e.g., inspecting register state |
| 78 | |
| 79 | ```text |
| 80 | (gdb) info registers |
| 81 | rax 0x0 0 |
| 82 | rbx 0x0 0 |
| 83 | rcx 0x40086 262278 |
| 84 | rdx 0x0 0 |
| 85 | rsi 0xffff960d4eea5010 -116491073990640 |
| 86 | rdi 0x0 0 |
| 87 | rbp 0x0 0x0 |
| 88 | rsp 0xfffff8015b3f5ec8 0xfffff8015b3f5ec8 |
| 89 | r8 0x0 0 |
| 90 | r9 0xffffffff 4294967295 |
| 91 | r10 0xfffff8015bfff1f0 -8790254554640 |
| 92 | r11 0x0 0 |
| 93 | r12 0xffffffff 4294967295 |
| 94 | ...etc... |
| 95 | ``` |
| 96 | |
| 97 | e.g., setting data breakpoints |
| 98 | |
| 99 | ```text |
| 100 | (gdb) awatch *0xfffff804683190e0 |
| 101 | Hardware access (read/write) watchpoint 1: *0xfffff804683190e0 |
| 102 | ``` |
| 103 | |
| 104 | e.g., single stepping |
| 105 | |
| 106 | ```text |
| 107 | 0xfffff8047a309686 in ?? () |
| 108 | (gdb) si |
| 109 | 0xfffff8047a309689 in ?? () |
| 110 | ``` |
| 111 | |
| 112 | You may find [this blog post](https://blog.mattjustice.com/2018/08/24/gdb-for-windbg-users/) |
| 113 | useful, as it includes a table of common `gdb` commands along with their WinDbg |
| 114 | counterparts. |
| 115 | |
| 116 | ## Connecting via WinDbg |
| 117 | |
| 118 | WinDbg doesn't understand the GDB Remote Serial Protocol directly, but |
| 119 | thankfully, some smart folks over on the WinDbg team have developed a GDB Remote |
| 120 | Serial Protocol <-> WinDbg translation layer! |
| 121 | |
| 122 | For more information, see |
| 123 | [Setting Up QEMU Kernel-Mode Debugging using EXDI](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-qemu-kernel-mode-debugging-using-exdi) |
| 124 | |
| 125 | Getting this working with OpenVMM or OpenHCL is as easy as following the guide, |
| 126 | except you'll need to [enable our debugger](#enabling-the-debugger) instead of |
| 127 | running QEMU. |
| 128 | |
| 129 | It's easiest to connect through the GUI. The steps are relatively simple: Open Windbgx -> File -> Attach to kernel -> EXDI. On the form, fill out: |
| 130 | - Target Type: `QEMU` |
| 131 | - Target Architecture: `X64` |
| 132 | - Target OS: `Windows` |
| 133 | - Image Screening heuristic size: `0xFFFE - NT` |
| 134 | - Gdb server and port: `<server>:<port>` e.g., `127.0.0.1:1337` (use whatever port you set above) |
| 135 | |
| 136 | ### Known WinDbg Bugs |
| 137 | - Hardware breakpoints are issued with [`ba`](https://learn.microsoft.com/en-us/windows-hardware/drivers/debuggercmds/ba--break-on-access-). The `Access Size` parameter is incorrectly multiplied by 8 when sent to the stub. Consequently, it _must_ be set to 1. |
| 138 | - Unlike GDB, WinDbg doesn't implicitly set software breakpoints via our offered write_addrs implementation. |
| 139 | --- |
| 140 | |
| 141 | |
| 142 | ## Supported Features |
| 143 | |
| 144 | At the time of writing (8/16/24) the debugger supports the following operations: |
| 145 | |
| 146 | - read/write guest memory |
| 147 | - read guest registers \* |
| 148 | - start/interrupt execution |
| 149 | - watchpoints |
| 150 | - hardware breakpoints |
| 151 | - single stepping |
| 152 | |
| 153 | ## TODO Features |
| 154 | |
| 155 | If you're looking for work, and want to improve the debugging experience for |
| 156 | everyone, consider implementing one or more of the following features: |
| 157 | |
| 158 | - \* reading _all_ guest registers, including fpu, xmm, and various key msrs |
| 159 | - software breakpoints: |
| 160 | - Intercept guest breakpoint exceptions into VTL2 |
| 161 | - writing guest registers |
| 162 | - exposing the OpenVMM interactive console via the |
| 163 | [`MonitorCmd`](https://docs.rs/gdbstub/latest/gdbstub/target/ext/monitor_cmd/trait.MonitorCmd.html) |
| 164 | interface |
| 165 | - Custom commands sent using `monitor` (gdb) / `.exdicmd` (WinDbg) |
| 166 | - e.g., being able to invoke `x device/to/inspect` directly from the debugger |
| 167 | - [any other features supported by the `gdbstub` library](https://github.com/daniel5151/gdbstub#debugging-features) |
| 168 | |