| 1 |
# vmtest — driving the installer in qemu |
| 2 |
|
| 3 |
Scripts that boot an Alloy ISO headless and work the wizard from outside the |
| 4 |
guest. They exist because `alloy install` is a ratatui program with no headless |
| 5 |
mode, so the only way to test the thing an installer medium actually does is to |
| 6 |
answer its questions and read the screen back. |
| 7 |
|
| 8 |
A medium can carry an answer sheet (`/usr/lib/alloy/answers.toml`, written from |
| 9 |
a per-host recipe) and the installer skips the steps it answers in full. That is |
| 10 |
a smaller set of questions to drive, never none: the account password and the |
| 11 |
LUKS passphrase are secrets and are never on a medium. `install_preseeded.py` is |
| 12 |
the scenario for a medium that carries one. |
| 13 |
|
| 14 |
Two halves, and they answer different questions. The tools (`run-vm.sh`, |
| 15 |
`vm.py`, `serial.py`, `ssh_pty.py`, `tui.py`, `qmp.py`) are for working a |
| 16 |
machine by hand, which is what most of this file is about. The scenarios |
| 17 |
(`install_drive.py`, `offline-first-boot.sh`) are the same work written down, |
| 18 |
so a defect that has been fixed once has something standing in front of it. |
| 19 |
|
| 20 |
Everything a run writes goes to `state/`, which is gitignored. Delete it to |
| 21 |
start clean. `build/build-iso.sh` clears `output/` at the start of every build, |
| 22 |
which is why the scratch directory is here instead. |
| 23 |
|
| 24 |
## Requirements |
| 25 |
|
| 26 |
`qemu-system-x86_64` with KVM, OVMF, and `swtpm` plus `swtpm-tools`. The |
| 27 |
software TPM is not optional for the encrypted path: `bootc install |
| 28 |
--block-setup tpm2-luks` enrols a key into one, and without a TPM it fails for |
| 29 |
a reason that has nothing to do with what is being tested. |
| 30 |
|
| 31 |
## Use |
| 32 |
|
| 33 |
./run-vm.sh live # boot output/install.iso (ALLOY_ISO= to override) |
| 34 |
./run-vm.sh installed # boot the target disk, no cdrom |
| 35 |
|
| 36 |
python3 vm.py hold 22 # hold the GRUB countdown open |
| 37 |
python3 vm.py key down down ret |
| 38 |
python3 vm.py type "some text" |
| 39 |
python3 vm.py shot name # screendump to state/name.png |
| 40 |
python3 serial.py "lsblk" 8 # run a command in the guest's serial root shell |
| 41 |
python3 ssh_pty.py ~/.ssh/id_ed25519 installer 30 |
| 42 |
python3 qmp.py cmd query-status |
| 43 |
python3 qmp.py wait RESET 900 '{"guest": true}' |
| 44 |
python3 tui.py 60 'step 1 of 6' ssh -tt -p 2222 installer@127.0.0.1 |
| 45 |
|
| 46 |
./offline-first-boot.sh # the scripted scenario; see below |
| 47 |
TARGET_BUS=nvme ./run-vm.sh live # a target that reports `tran: nvme` |
| 48 |
python3 install_preseeded.py # install from a medium with an answer sheet |
| 49 |
|
| 50 |
`KEYBOARD` decides what keyboards the machine has, and exists because the USB |
| 51 |
gate's bench tests need hardware no machine here can be. `usr/bin/alloy-usb-gate` |
| 52 |
suspends enforcement whenever the count of `ID_INPUT_KEYBOARD` devices reaches |
| 53 |
zero, and every physical machine in this tree has an i8042 keyboard soldered to |
| 54 |
it that USBGuard has no jurisdiction over, so the count never reaches zero on |
| 55 |
real hardware. `ps2` is q35's own i8042 controller and the default. `none` is |
| 56 |
`i8042=off`: no keyboard of any kind, which is the machine the gate is supposed |
| 57 |
to open for at boot. `usb` is `i8042=off` plus a USB keyboard on an xHCI |
| 58 |
controller, so the only keyboard is one USBGuard can take away; its `id=kbd0` is |
| 59 |
what `qmp.py cmd device_del '{"id": "kbd0"}'` unplugs, which is the third bench |
| 60 |
test and cannot be done from inside the guest. |
| 61 |
|
| 62 |
KEYBOARD=none ./run-vm.sh installed # zero keyboards |
| 63 |
KEYBOARD=usb ./run-vm.sh installed # one, and it is removable |
| 64 |
|
| 65 |
`TARGET_BUS` decides how the target disk is attached, and it is not a |
| 66 |
performance knob. A recipe whose disk rule is `single-internal-nvme` cannot be |
| 67 |
exercised against a virtio disk, whose lsblk `tran` is null, and a rule that |
| 68 |
does not match falls back to asking, which looks exactly like the feature being |
| 69 |
broken. Default stays `virtio`, because every scenario written before this one |
| 70 |
assumes `vda`. |
| 71 |
|
| 72 |
`vm.py shot` converts qemu's PPM to PNG with nothing but zlib and struct, so |
| 73 |
the installer screen can be read directly with no image tooling installed. |
| 74 |
`serial.py` reaches a root shell only on GRUB entry 3, which is the debug |
| 75 |
entry; the default entry runs the wizard and has no shell. |
| 76 |
|
| 77 |
`qmp.py` is the machine monitor, next to `vm.py`'s human one rather than |
| 78 |
instead of it. HMP is right for sendkey and screendump; QMP is the only one |
| 79 |
that can say WHEN something happened inside the guest, because it emits events. |
| 80 |
`set_link` lives there too: taking the guest's route away from outside is the |
| 81 |
only way that does not depend on the guest cooperating, and a `RESET` with |
| 82 |
`guest: true` is how "the machine rebooted itself" becomes a thing to wait for |
| 83 |
rather than a thing to infer from the clock. |
| 84 |
|
| 85 |
`tui.py` reads the screen rather than the bytes, and the difference is not a |
| 86 |
refinement. ratatui redraws only the cells that changed, so `step 1 of 6` |
| 87 |
becoming `step 2 of 6` puts a single `2` on the wire behind a cursor move: a |
| 88 |
script grepping the stream finds the title it is waiting for exactly once, on |
| 89 |
the first frame where everything is written, and then silently never again. |
| 90 |
`ssh_pty.py` captures the stream and is the right tool for looking at one; |
| 91 |
anything that has to make a decision from what is on screen wants this. It |
| 92 |
carries a `--self-test` for the parser, for the same reason |
| 93 |
`check-installed.sh` carries one: a wrong parse still produces a screen. |
| 94 |
|
| 95 |
## Things that cost an afternoon to learn |
| 96 |
|
| 97 |
**Reset between runs by deleting `state/target.qcow2` and `state/OVMF_VARS.fd` |
| 98 |
together.** Leaving the firmware variables behind boots the half-written disk |
| 99 |
to a `grub>` prompt. Delete `state/tpm/` as well to simulate a cleared TPM. |
| 100 |
|
| 101 |
**Catch GRUB before the countdown expires.** The menu holds for five seconds |
| 102 |
from a cold boot, which is roughly when the harness is still starting. Reset |
| 103 |
with `vm.py cmd system_reset` and run `vm.py hold` immediately; any keypress |
| 104 |
stops the countdown, and `hold` sends one every 200ms. |
| 105 |
|
| 106 |
**Screenshot before pressing return in the GRUB menu.** Missing the countdown |
| 107 |
boots the default entry, and there is then no shell to recover through. |
| 108 |
|
| 109 |
**The activity light blinks, so two screendumps of an unchanged screen differ.** |
| 110 |
Comparing frames to detect progress means ignoring the header rows. |
| 111 |
|
| 112 |
**The erase confirmation takes a second return.** The review footer reads |
| 113 |
`enter install`, but the first return raises a modal. A script that sends one |
| 114 |
keystroke and waits sits there forever looking like a slow install. |
| 115 |
|
| 116 |
**Watch the target's size, not the clock.** A real install passes a gigabyte in |
| 117 |
the first minute; a qcow2 still at its created size means nothing has started. |
| 118 |
|
| 119 |
**The recovery phrase is legible in a screendump.** This was believed to need a |
| 120 |
person, which is why the encrypted path went untested for so long. It does not. |
| 121 |
|
| 122 |
**The installed machine's login shell is nushell, so a remote command is not a |
| 123 |
shell command.** ssh hands what you give it to the login shell, and that is |
| 124 |
`/usr/bin/nu`. `... 2>&1` is a parse error there (`use 'out+err>'`), `cat > |
| 125 |
file` reads the `>` as an argument, and `scp` fails outright because it runs |
| 126 |
its far end through the login shell too. Two of those three read as the machine |
| 127 |
being broken rather than as the command being wrong. Route everything through a |
| 128 |
POSIX shell explicitly, and prefer `ssh <host> 'sh -s' < script`: the script is |
| 129 |
then written once in its own syntax with nothing to quote through nushell. Use |
| 130 |
`sh -c '...'` only when stdin is carrying something else. |
| 131 |
|
| 132 |
**`run-vm.sh` re-creates the software TPM whenever swtpm is not running, and |
| 133 |
that silently breaks an installed disk that unlocks with one.** The setup call |
| 134 |
carries `--overwrite`, and it fires whenever `state/tpm/swtpm-sock` is absent — |
| 135 |
which is every time the swtpm daemon has been stopped, not only on a first run. |
| 136 |
A new TPM has a new seed, so the sealed key in the LUKS header can no longer be |
| 137 |
loaded, and the guest asks for the passphrase instead. What it prints is |
| 138 |
`Failed to unseal secret using TPM2: State not recoverable` with |
| 139 |
`Esys_Load() ... ErrorCode (0x0000018b)` above it, in the journal of the boot |
| 140 |
that prompted. It reads like a broken enrolment rather than a harness that |
| 141 |
changed the hardware underneath it. Delete `state/tpm/` deliberately to simulate |
| 142 |
a cleared TPM; do not let it happen by accident. |
| 143 |
|
| 144 |
**Re-enrolling a TPM2 keyslot takes two commands, not one.** |
| 145 |
`systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto <dev>` answers |
| 146 |
`This PCR set is already enrolled, executing no operation` and changes nothing, |
| 147 |
because it compares against the existing enrolment before it wipes. Wipe and |
| 148 |
enrol separately, and read the output: the pair that works prints `Wiped slot N` |
| 149 |
and then `New TPM2 token enrolled as key slot N`. |
| 150 |
|
| 151 |
**A machine with `KEYBOARD=none` cannot answer a LUKS passphrase prompt.** |
| 152 |
There is no input device for `vm.py type` to reach, so an encrypted target is |
| 153 |
unreachable in that mode and the run stalls at the prompt with no way forward. |
| 154 |
Bench-testing the gate against an encrypted disk therefore depends on the TPM |
| 155 |
path working. `install_drive.py` leaves encryption off by default, and a target |
| 156 |
installed that way is the one to use for keyboard-shaped tests. |
| 157 |
|
| 158 |
**The medium's ssh installer needs privilege it does not get on its own.** |
| 159 |
`alloy install` does not escalate, and an ssh session lands unprivileged, so the |
| 160 |
wizard draws and answers every step and then fails on the first command that |
| 161 |
writes a disk. The medium runs the wizard under `run0`, with a polkit grant for |
| 162 |
the one action run0 asks for. `VMTEST_VIA=serial` needs none of that. |
| 163 |
|
| 164 |
## Check the result, not only the screen |
| 165 |
|
| 166 |
An install that finishes is not an install that worked. `bootc container lint` |
| 167 |
answers for the image, and `build/check-installed.sh` answers for what the |
| 168 |
installer wrote afterwards. Nothing else does: a mislabelled `/etc` passes every |
| 169 |
image check and surfaces only when something needs `DynamicUser`. |
| 170 |
|
| 171 |
Run it on the installed guest once it is up: |
| 172 |
|
| 173 |
ssh -p 2222 -i ~/.ssh/id_ed25519 <user>@127.0.0.1 'sudo bash -s' \ |
| 174 |
< ../check-installed.sh |
| 175 |
|
| 176 |
`run-vm.sh` forwards the guest's ssh to :2222 on both `live` and `installed`, |
| 177 |
which is the same port `ssh_pty.py` uses. |
| 178 |
|
| 179 |
Exit 0 is a labelled machine whose `DynamicUser` units start, 1 is a defect |
| 180 |
with the repair printed, and 3 means the check could not run — not root, no |
| 181 |
`restorecon`, or an install from the `selinux=0` GRUB entry, which deliberately |
| 182 |
produces an unlabelled machine. |
| 183 |
|
| 184 |
`--policy` is the other question a booted machine can answer and the build host |
| 185 |
cannot: what polkit's implicit defaults actually are for every action the |
| 186 |
console goes through. It needs no root, and wiki `alloy-privilege` is the table |
| 187 |
it checks against. |
| 188 |
|
| 189 |
## Checking the firewall |
| 190 |
|
| 191 |
The one thing a container build cannot answer, and it needs the guest to be |
| 192 |
reachable from outside. User-mode networking gives the guest no inbound path |
| 193 |
except a `hostfwd`, so a port the host simply cannot reach proves nothing: |
| 194 |
`run-vm.sh` forwards :2223 to guest :5555 for exactly this, next to ssh on |
| 195 |
:2222. Override the host side with `PROBE_PORT=` if 2223 is taken. |
| 196 |
|
| 197 |
Listen inside the guest, then knock from the host: |
| 198 |
|
| 199 |
python3 serial.py 'socat TCP-LISTEN:5555,reuseaddr,fork EXEC:/bin/echo &' |
| 200 |
python3 -c 'import socket;s=socket.socket();s.settimeout(6);s.connect(("127.0.0.1",2223));print(s.recv(64))' |
| 201 |
|
| 202 |
**A blocked port reads as a timeout, not as a refusal.** qemu accepts the host |
| 203 |
side of a forward before it knows what the guest will do, so the connect always |
| 204 |
succeeds and the drop shows up as silence. Read the data, never the connect. |
| 205 |
|
| 206 |
**Prove the silence is the firewall, or it proves nothing.** A listener that was |
| 207 |
not running looks identical. Open the port, knock again, close it: |
| 208 |
|
| 209 |
python3 serial.py 'firewall-cmd --add-port=5555/tcp' # data arrives |
| 210 |
python3 serial.py 'firewall-cmd --remove-port=5555/tcp' # silence returns |
| 211 |
|
| 212 |
On a server-profile medium the live ISO and the machine it installs differ by |
| 213 |
one service on purpose: `alloy-installer-firewall.service` adds mdns at runtime |
| 214 |
so a headless install is findable at `<name>.local`, and the installed system |
| 215 |
must not have it. `firewall-cmd --list-services` on both is the check; |
| 216 |
`--permanent --list-services` on the live medium shows the runtime add is not |
| 217 |
written anywhere. |
| 218 |
|
| 219 |
## The scripted scenario: an offline first boot |
| 220 |
|
| 221 |
`./offline-first-boot.sh` installs a machine, takes its route away before the |
| 222 |
first boot, and checks that the console gets laid down anyway. |
| 223 |
|
| 224 |
It exists because that defect shipped. `alloy-layer-components.service` |
| 225 |
installs the console and terminal from `/usr/share/alloy/rpm`, a `file://` |
| 226 |
repo carried on the medium precisely so an offline install produces a working |
| 227 |
machine, and it did not: rpm-ostree refreshes metadata for every ENABLED repo |
| 228 |
before it depsolves, so with Fedora's four enabled a first boot with no name |
| 229 |
resolution failed on a mirrorlist it did not need, and the machine came up with |
| 230 |
no console at all. Fixed in `aa0dacd` by fencing the transaction to the carried |
| 231 |
repo (`usr/bin/alloy-layer-repos`), verified by hand, and until this script |
| 232 |
nothing here would have caught it coming back. |
| 233 |
|
| 234 |
Red before `aa0dacd`, green after. The failing shape is specific: the unit |
| 235 |
fails, so it never reaches its `systemctl reboot`, so no `RESET` arrives and |
| 236 |
`/usr/bin/alloy` is absent afterwards. |
| 237 |
|
| 238 |
The installer is driven over the medium's own headless ssh account, which is |
| 239 |
the route a person installing a screenless machine takes. So the ISO needs a |
| 240 |
baked pubkey matching the key given here: |
| 241 |
|
| 242 |
build/build-iso.sh --build-arg PROFILE=server --build-arg BROWSER=none \ |
| 243 |
--build-arg ALLOY_SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)" |
| 244 |
build/vmtest/offline-first-boot.sh |
| 245 |
|
| 246 |
That key does two jobs: it is the installer session's only credential, and it is |
| 247 |
what the wizard puts in the new machine's `authorized_keys`, which is how the |
| 248 |
assertions get in afterwards. `VMTEST_VIA=serial` drives GRUB's debug shell |
| 249 |
instead and needs only the second. |
| 250 |
|
| 251 |
GRUB is navigated by reading the menu off the serial console and checking the |
| 252 |
mark before pressing return, so neither of the countdown warnings above applies |
| 253 |
to this script. |
| 254 |
|
| 255 |
Four things are asserted, and the last is folded in rather than separate |
| 256 |
because a scenario that boots a fresh machine is already holding everything |
| 257 |
`check-installed.sh` needs: |
| 258 |
|
| 259 |
1. the machine rebooted itself, which is the unit's last act |
| 260 |
2. `/usr/bin/alloy` exists |
| 261 |
3. the enabled repo set is `alloy-local` plus Fedora's four, so the fence was |
| 262 |
lowered again and `alloy pkg` will still find Fedora packages |
| 263 |
4. `check-installed.sh` is clean on the result |
| 264 |
|
| 265 |
Exit 0 is a machine that survived, 1 is one that did not with the evidence |
| 266 |
printed, and 3 means the run could not reach a verdict. |
| 267 |
|
| 268 |
**The route is taken away from the monitor, not from inside.** `ip link set |
| 269 |
enp0s2 down` travels over the ssh session it arrived on and kills it, which is |
| 270 |
why the hand run needed a detached `setsid` script that brought the interface |
| 271 |
back up at the end. From outside, the link is a property of the device and the |
| 272 |
guest gets no say. It goes back up only after the reboot, where it cannot |
| 273 |
affect the answer: the unit's `ConditionPathExists=!/usr/bin/alloy` is already |
| 274 |
false on a machine that has a console. |
| 275 |
|
| 276 |
**A machine that has already been first-booted is not a run of this test.** The |
| 277 |
unit is a no-op there, so no reboot happens and no reboot SHOULD happen. The |
| 278 |
script reports that as exit 3 rather than as a regression, which is the one |
| 279 |
thing `--keep-state` could otherwise turn into a false alarm. |
| 280 |
|
| 281 |
## What it caught |
| 282 |
|
| 283 |
Two defects in `crates/alloy/src/install.rs`: an unmounted boot partition that |
| 284 |
made ostree report no deployment on every encrypted install, and bootc's |
| 285 |
`headless=true` karg that left the enrolled passphrase untypeable at boot. |
| 286 |
Neither is visible without booting the result. |
| 287 |
|
| 288 |
Wiki `alloy-build-notes` carries the longer form, including how to read a |
| 289 |
failed target before the installer's recovery erases it. |
| 290 |
|