# vmtest — driving the installer in qemu

Scripts that boot an Alloy ISO headless and work the wizard from outside the
guest. They exist because `alloy install` is a ratatui program with no headless
mode, so the only way to test the thing an installer medium actually does is to
answer its questions and read the screen back.

A medium can carry an answer sheet (`/usr/lib/alloy/answers.toml`, written from
a per-host recipe) and the installer skips the steps it answers in full. That is
a smaller set of questions to drive, never none: the account password and the
LUKS passphrase are secrets and are never on a medium. `install_preseeded.py` is
the scenario for a medium that carries one.

Two halves, and they answer different questions. The tools (`run-vm.sh`,
`vm.py`, `serial.py`, `ssh_pty.py`, `tui.py`, `qmp.py`) are for working a
machine by hand, which is what most of this file is about. The scenarios
(`install_drive.py`, `offline-first-boot.sh`) are the same work written down,
so a defect that has been fixed once has something standing in front of it.

Everything a run writes goes to `state/`, which is gitignored. Delete it to
start clean. `build/build-iso.sh` clears `output/` at the start of every build,
which is why the scratch directory is here instead.

## Requirements

`qemu-system-x86_64` with KVM, OVMF, and `swtpm` plus `swtpm-tools`. The
software TPM is not optional for the encrypted path: `bootc install
--block-setup tpm2-luks` enrols a key into one, and without a TPM it fails for
a reason that has nothing to do with what is being tested.

## Use

    ./run-vm.sh live            # boot output/install.iso (ALLOY_ISO= to override)
    ./run-vm.sh installed       # boot the target disk, no cdrom

    python3 vm.py hold 22       # hold the GRUB countdown open
    python3 vm.py key down down ret
    python3 vm.py type "some text"
    python3 vm.py shot name     # screendump to state/name.png
    python3 serial.py "lsblk" 8 # run a command in the guest's serial root shell
    python3 ssh_pty.py ~/.ssh/id_ed25519 installer 30
    python3 qmp.py cmd query-status
    python3 qmp.py wait RESET 900 '{"guest": true}'
    python3 tui.py 60 'step 1 of 6' ssh -tt -p 2222 installer@127.0.0.1

    ./offline-first-boot.sh     # the scripted scenario; see below
    TARGET_BUS=nvme ./run-vm.sh live    # a target that reports `tran: nvme`
    python3 install_preseeded.py        # install from a medium with an answer sheet

`KEYBOARD` decides what keyboards the machine has, and exists because the USB
gate's bench tests need hardware no machine here can be. `usr/bin/alloy-usb-gate`
suspends enforcement whenever the count of `ID_INPUT_KEYBOARD` devices reaches
zero, and every physical machine in this tree has an i8042 keyboard soldered to
it that USBGuard has no jurisdiction over, so the count never reaches zero on
real hardware. `ps2` is q35's own i8042 controller and the default. `none` is
`i8042=off`: no keyboard of any kind, which is the machine the gate is supposed
to open for at boot. `usb` is `i8042=off` plus a USB keyboard on an xHCI
controller, so the only keyboard is one USBGuard can take away; its `id=kbd0` is
what `qmp.py cmd device_del '{"id": "kbd0"}'` unplugs, which is the third bench
test and cannot be done from inside the guest.

    KEYBOARD=none ./run-vm.sh installed   # zero keyboards
    KEYBOARD=usb  ./run-vm.sh installed   # one, and it is removable

`TARGET_BUS` decides how the target disk is attached, and it is not a
performance knob. A recipe whose disk rule is `single-internal-nvme` cannot be
exercised against a virtio disk, whose lsblk `tran` is null, and a rule that
does not match falls back to asking, which looks exactly like the feature being
broken. Default stays `virtio`, because every scenario written before this one
assumes `vda`.

`vm.py shot` converts qemu's PPM to PNG with nothing but zlib and struct, so
the installer screen can be read directly with no image tooling installed.
`serial.py` reaches a root shell only on GRUB entry 3, which is the debug
entry; the default entry runs the wizard and has no shell.

`qmp.py` is the machine monitor, next to `vm.py`'s human one rather than
instead of it. HMP is right for sendkey and screendump; QMP is the only one
that can say WHEN something happened inside the guest, because it emits events.
`set_link` lives there too: taking the guest's route away from outside is the
only way that does not depend on the guest cooperating, and a `RESET` with
`guest: true` is how "the machine rebooted itself" becomes a thing to wait for
rather than a thing to infer from the clock.

`tui.py` reads the screen rather than the bytes, and the difference is not a
refinement. ratatui redraws only the cells that changed, so `step 1 of 6`
becoming `step 2 of 6` puts a single `2` on the wire behind a cursor move: a
script grepping the stream finds the title it is waiting for exactly once, on
the first frame where everything is written, and then silently never again.
`ssh_pty.py` captures the stream and is the right tool for looking at one;
anything that has to make a decision from what is on screen wants this. It
carries a `--self-test` for the parser, for the same reason
`check-installed.sh` carries one: a wrong parse still produces a screen.

## Things that cost an afternoon to learn

**Reset between runs by deleting `state/target.qcow2` and `state/OVMF_VARS.fd`
together.** Leaving the firmware variables behind boots the half-written disk
to a `grub>` prompt. Delete `state/tpm/` as well to simulate a cleared TPM.

**Catch GRUB before the countdown expires.** The menu holds for five seconds
from a cold boot, which is roughly when the harness is still starting. Reset
with `vm.py cmd system_reset` and run `vm.py hold` immediately; any keypress
stops the countdown, and `hold` sends one every 200ms.

**Screenshot before pressing return in the GRUB menu.** Missing the countdown
boots the default entry, and there is then no shell to recover through.

**The activity light blinks, so two screendumps of an unchanged screen differ.**
Comparing frames to detect progress means ignoring the header rows.

**The erase confirmation takes a second return.** The review footer reads
`enter install`, but the first return raises a modal. A script that sends one
keystroke and waits sits there forever looking like a slow install.

**Watch the target's size, not the clock.** A real install passes a gigabyte in
the first minute; a qcow2 still at its created size means nothing has started.

**The recovery phrase is legible in a screendump.** This was believed to need a
person, which is why the encrypted path went untested for so long. It does not.

**The installed machine's login shell is nushell, so a remote command is not a
shell command.** ssh hands what you give it to the login shell, and that is
`/usr/bin/nu`. `... 2>&1` is a parse error there (`use 'out+err>'`), `cat >
file` reads the `>` as an argument, and `scp` fails outright because it runs
its far end through the login shell too. Two of those three read as the machine
being broken rather than as the command being wrong. Route everything through a
POSIX shell explicitly, and prefer `ssh <host> 'sh -s' < script`: the script is
then written once in its own syntax with nothing to quote through nushell. Use
`sh -c '...'` only when stdin is carrying something else.

**`run-vm.sh` re-creates the software TPM whenever swtpm is not running, and
that silently breaks an installed disk that unlocks with one.** The setup call
carries `--overwrite`, and it fires whenever `state/tpm/swtpm-sock` is absent —
which is every time the swtpm daemon has been stopped, not only on a first run.
A new TPM has a new seed, so the sealed key in the LUKS header can no longer be
loaded, and the guest asks for the passphrase instead. What it prints is
`Failed to unseal secret using TPM2: State not recoverable` with
`Esys_Load() ... ErrorCode (0x0000018b)` above it, in the journal of the boot
that prompted. It reads like a broken enrolment rather than a harness that
changed the hardware underneath it. Delete `state/tpm/` deliberately to simulate
a cleared TPM; do not let it happen by accident.

**Re-enrolling a TPM2 keyslot takes two commands, not one.**
`systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto <dev>` answers
`This PCR set is already enrolled, executing no operation` and changes nothing,
because it compares against the existing enrolment before it wipes. Wipe and
enrol separately, and read the output: the pair that works prints `Wiped slot N`
and then `New TPM2 token enrolled as key slot N`.

**A machine with `KEYBOARD=none` cannot answer a LUKS passphrase prompt.**
There is no input device for `vm.py type` to reach, so an encrypted target is
unreachable in that mode and the run stalls at the prompt with no way forward.
Bench-testing the gate against an encrypted disk therefore depends on the TPM
path working. `install_drive.py` leaves encryption off by default, and a target
installed that way is the one to use for keyboard-shaped tests.

**The medium's ssh installer needs privilege it does not get on its own.**
`alloy install` does not escalate, and an ssh session lands unprivileged, so the
wizard draws and answers every step and then fails on the first command that
writes a disk. The medium runs the wizard under `run0`, with a polkit grant for
the one action run0 asks for. `VMTEST_VIA=serial` needs none of that.

## Check the result, not only the screen

An install that finishes is not an install that worked. `bootc container lint`
answers for the image, and `build/check-installed.sh` answers for what the
installer wrote afterwards. Nothing else does: a mislabelled `/etc` passes every
image check and surfaces only when something needs `DynamicUser`.

Run it on the installed guest once it is up:

    ssh -p 2222 -i ~/.ssh/id_ed25519 <user>@127.0.0.1 'sudo bash -s' \
        < ../check-installed.sh

`run-vm.sh` forwards the guest's ssh to :2222 on both `live` and `installed`,
which is the same port `ssh_pty.py` uses.

Exit 0 is a labelled machine whose `DynamicUser` units start, 1 is a defect
with the repair printed, and 3 means the check could not run — not root, no
`restorecon`, or an install from the `selinux=0` GRUB entry, which deliberately
produces an unlabelled machine.

`--policy` is the other question a booted machine can answer and the build host
cannot: what polkit's implicit defaults actually are for every action the
console goes through. It needs no root, and wiki `alloy-privilege` is the table
it checks against.

## Checking the firewall

The one thing a container build cannot answer, and it needs the guest to be
reachable from outside. User-mode networking gives the guest no inbound path
except a `hostfwd`, so a port the host simply cannot reach proves nothing:
`run-vm.sh` forwards :2223 to guest :5555 for exactly this, next to ssh on
:2222. Override the host side with `PROBE_PORT=` if 2223 is taken.

Listen inside the guest, then knock from the host:

    python3 serial.py 'socat TCP-LISTEN:5555,reuseaddr,fork EXEC:/bin/echo &'
    python3 -c 'import socket;s=socket.socket();s.settimeout(6);s.connect(("127.0.0.1",2223));print(s.recv(64))'

**A blocked port reads as a timeout, not as a refusal.** qemu accepts the host
side of a forward before it knows what the guest will do, so the connect always
succeeds and the drop shows up as silence. Read the data, never the connect.

**Prove the silence is the firewall, or it proves nothing.** A listener that was
not running looks identical. Open the port, knock again, close it:

    python3 serial.py 'firewall-cmd --add-port=5555/tcp'      # data arrives
    python3 serial.py 'firewall-cmd --remove-port=5555/tcp'   # silence returns

On a server-profile medium the live ISO and the machine it installs differ by
one service on purpose: `alloy-installer-firewall.service` adds mdns at runtime
so a headless install is findable at `<name>.local`, and the installed system
must not have it. `firewall-cmd --list-services` on both is the check;
`--permanent --list-services` on the live medium shows the runtime add is not
written anywhere.

## The scripted scenario: an offline first boot

`./offline-first-boot.sh` installs a machine, takes its route away before the
first boot, and checks that the console gets laid down anyway.

It exists because that defect shipped. `alloy-layer-components.service`
installs the console and terminal from `/usr/share/alloy/rpm`, a `file://`
repo carried on the medium precisely so an offline install produces a working
machine, and it did not: rpm-ostree refreshes metadata for every ENABLED repo
before it depsolves, so with Fedora's four enabled a first boot with no name
resolution failed on a mirrorlist it did not need, and the machine came up with
no console at all. Fixed in `aa0dacd` by fencing the transaction to the carried
repo (`usr/bin/alloy-layer-repos`), verified by hand, and until this script
nothing here would have caught it coming back.

Red before `aa0dacd`, green after. The failing shape is specific: the unit
fails, so it never reaches its `systemctl reboot`, so no `RESET` arrives and
`/usr/bin/alloy` is absent afterwards.

The installer is driven over the medium's own headless ssh account, which is
the route a person installing a screenless machine takes. So the ISO needs a
baked pubkey matching the key given here:

    build/build-iso.sh --build-arg PROFILE=server --build-arg BROWSER=none \
      --build-arg ALLOY_SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)"
    build/vmtest/offline-first-boot.sh

That key does two jobs: it is the installer session's only credential, and it is
what the wizard puts in the new machine's `authorized_keys`, which is how the
assertions get in afterwards. `VMTEST_VIA=serial` drives GRUB's debug shell
instead and needs only the second.

GRUB is navigated by reading the menu off the serial console and checking the
mark before pressing return, so neither of the countdown warnings above applies
to this script.

Four things are asserted, and the last is folded in rather than separate
because a scenario that boots a fresh machine is already holding everything
`check-installed.sh` needs:

  1. the machine rebooted itself, which is the unit's last act
  2. `/usr/bin/alloy` exists
  3. the enabled repo set is `alloy-local` plus Fedora's four, so the fence was
     lowered again and `alloy pkg` will still find Fedora packages
  4. `check-installed.sh` is clean on the result

Exit 0 is a machine that survived, 1 is one that did not with the evidence
printed, and 3 means the run could not reach a verdict.

**The route is taken away from the monitor, not from inside.** `ip link set
enp0s2 down` travels over the ssh session it arrived on and kills it, which is
why the hand run needed a detached `setsid` script that brought the interface
back up at the end. From outside, the link is a property of the device and the
guest gets no say. It goes back up only after the reboot, where it cannot
affect the answer: the unit's `ConditionPathExists=!/usr/bin/alloy` is already
false on a machine that has a console.

**A machine that has already been first-booted is not a run of this test.** The
unit is a no-op there, so no reboot happens and no reboot SHOULD happen. The
script reports that as exit 3 rather than as a regression, which is the one
thing `--keep-state` could otherwise turn into a false alarm.

## What it caught

Two defects in `crates/alloy/src/install.rs`: an unmounted boot partition that
made ostree report no deployment on every encrypted install, and bootc's
`headless=true` karg that left the enrolled passphrase untypeable at boot.
Neither is visible without booting the result.

Wiki `alloy-build-notes` carries the longer form, including how to read a
failed target before the installer's recovery erases it.