# layertest — what a hotfix is allowed to do to an installed machine Alloy is distributed as a builder rather than as an image, so there is no way to hand a running machine a fix short of "rebuild the ISO and write a drive". The proposed answer is a signed RPM repo carrying only our own components, layered with `rpm-ostree` (GoingsOn task `d866e125`). That answer rests on one assumption: that a package layered today is still there, and still correct, after the base moves tomorrow. This harness measures that assumption instead of trusting it. It is not a test suite and nothing runs it automatically. It exists to be re-run when rpm-ostree or bootc moves, because every result below is a behaviour of those two and not of anything in this repo. ## What it found Measured on rpm-ostree 2026.1, bootc 1.16.3, libostree 2026.2, against a `quay.io/fedora/fedora-bootc:43` base. **`bootc upgrade` refuses to run on a layered deployment.** It does not discard the layer and does not keep it. It errors and does nothing: error: Upgrading: Deployment contains local rpm-ostree modifications; cannot upgrade via bootc. You can run `rpm-ostree reset` to undo the modifications. The two mechanisms are mutually exclusive. `rpm-ostree upgrade` is the verb that works on a layered system, and it pulls the same container image, so anything shipping a hotfix channel has to move machines off `bootc upgrade`. **A component already in the base cannot be replaced from a repo.** With the component installed as a base RPM at 0.0.1 and 0.0.2 in the repo: - `rpm-ostree install` fails to depsolve, `cannot install both ... from @System`. - `rpm-ostree override replace` reports `Inactive base replacements` and writes no new commit, in both of its forms. They differ only in what they leave behind, and neither difference helps: given a local RPM or a URL, the request is recorded in `requested-base-local-replacements` and survives a reboot as a silent no-op, so the machine reports a hotfix it is not running. Given `--experimental --from repo=`, nothing is recorded at all once the machine reboots, so the request disappears without ever having done anything. - `override remove` on a base package does commit, but pairing it with `--install` fails the same depsolve. Reproduce with variant `r1`. **A component that is an unowned file cannot be layered over at all.** A `COPY` line putting a binary into `/usr/bin` leaves it owned by no package, and this measurement is why Alloy does not ship one. Layering an RPM that carries the same path dies in checkout rather than in depsolve: error: Checkout alloy-demo-0.0.2: Hardlinking ... to alloy-demo: File exists Reproduce with variant `l1`. **The shape that does work, and where it breaks.** If the base does not carry the component at all, the hotfix is a plain layered package and applies cleanly. Across `rpm-ostree upgrade`: | the base moves to | what happens to the layer | |---|---| | a base still without the component (`n2`) | survives, re-applies, clean | | a base carrying it at the same version (`n3`) | goes `Inactive requests: already provided by`, drops out of `packages`, stays in `requested-packages`. Self-healing | | a base carrying a **newer** version (`n4`) | **the machine stops updating**, `cannot install both 0.0.2 from alloy-demo and 0.0.3 from @System`, and every later upgrade fails too | That last row is the normal life of a hotfix: ship it out of band, fold it into the next image at a higher version, and every machine that took it wedges. Recovery is `rpm-ostree uninstall` and then upgrade, after which `bootc upgrade` works again, but a user has no way to know that is what happened. ## The wedge is avoidable Measured after the above, on the same versions. The rule is that **our layers are disposable**: never carried across an upgrade, dropped before it, and re-applied afterwards only if the base still lacks the fix. If no layer and no base copy are ever in the same transaction, the depsolve conflict cannot arise. It holds. From a machine on `n1` holding a layered `alloy-demo-0.0.2`, against `n4` (base carrying 0.0.3), `case.sh n4 --drain`: Removed: alloy-demo-0.0.2-1.fc43.x86_64 Upgraded: alloy-demo 0.0.2-1.fc43 -> 0.0.3-1.fc43 base-mark: n4 binary: alloy-demo 0.0.3 no layers **One reboot, not two.** The uninstall and the upgrade each stage a deployment and the second composes onto the first, so both land in one boot. The rollback deployment still holds the layered state, which is what a rollback is for. **`bootc upgrade` consents again.** With no local modifications left, the machine is back on the bootc path rather than stranded on `rpm-ostree upgrade` forever. **The sharp edge.** A request is recorded under whatever string was typed to install it, so a package layered by full NEVRA cannot be removed by its bare name: # rpm-ostree uninstall alloy-demo error: Package/capability 'alloy-demo' is not currently requested with `alloy-demo-0.0.2-1.fc43.x86_64` sitting in `requested-packages` the whole time. That reads as "there was nothing to do" rather than as a failure, so a drain written against the package name silently does not drain and the machine wedges exactly as if the rule had never been applied. Anything implementing this has to read `requested-packages` and uninstall the exact strings it finds; `readstate.py --requested ` is that lookup, and `--drain` uses it. Not `rpm-ostree reset`, though bootc's own error message suggests it and it would work here. Reset drops every layered package including ones the user layered themselves, and those are not ours to remove. ## The carry shape, and it is the one Alloy takes The image carries our RPMs **as files**, in a `file://` repo under `/usr/share/alloy/rpm`, and does not install them. The component is still absent from `@System`, so it layers exactly as in the `none` cases, but the package it layers from travels inside the image. Variants `c1` (carries 0.0.2) and `c2` (carries 0.0.3). Nothing has to be copied anywhere at install time. The ISO is the image, so the ISO carries the repo; the installed machine is the image, so it carries the repo too. That is the whole delivery mechanism. **Layering works with no network.** With the network repo `enabled=0`, from a `c1` machine: # rpm-ostree install -y --idempotent alloy-demo Added: alloy-demo-0.0.2-1.fc43.x86_64 Requested under the **bare name**, because that is what was typed. Worth noticing next to the NEVRA trap recorded below: install by name and the drain can uninstall by name. **A base move does not wedge it.** `c1` to `c2`, base carrying 0.0.3 against a layer at 0.0.2, no drain, plain `rpm-ostree upgrade`: it succeeds. This is the whole reason to prefer this shape over shipping the component as a base RPM. `@System` never holds the component, so the conflict that permanently wedges the `n4` case has nothing to arise between. **But the layer does not follow the image.** After that upgrade the machine is on base `c2` and still running `alloy-demo 0.0.2`, while the base beside it carries 0.0.3. rpm-ostree does not re-resolve a satisfied request just because a repo behind it moved. Shipping a new console inside a new image therefore reaches nobody on its own, which is the opposite of what it looks like. **What advances it is a drop and reinstall**, in one boot: # rpm-ostree uninstall alloy-demo && rpm-ostree install -y alloy-demo Upgraded: alloy-demo 0.0.2-1.fc43 -> 0.0.3-1.fc43 The drain rule holds here for a second reason: it is the only thing that makes a component track the image it shipped with. `alloy update` owns it. **`bootc upgrade` refuses permanently, not transiently.** An Alloy machine always carries a layer, so it always has "local rpm-ostree modifications". `alloy update` owns `rpm-ostree upgrade` for the life of the machine, and moving users off `bootc upgrade` is a permanent condition rather than a step in a migration. ## `bootc switch` is not `bootc upgrade` The paragraph above is about `upgrade`, and reads as though the whole bootc path were closed to a layered machine. It is not, and the difference matters because **`bootc switch` is the command Alloy actually prints**: it is the middle line of `Staleness::REMEDY` in `crates/alloy/src/stale.rs` and the adoption step in docs/IMAGE.md. Measured on the same versions, from a `c1` machine holding a layered `alloy-demo`, switching to `c2`: # bootc upgrade error: Upgrading: Deployment contains local rpm-ostree modifications; cannot upgrade via bootc. # bootc switch --transport registry 10.0.2.2:5000/alloy-layertest:c2 Queued for next boot: 10.0.2.2:5000/alloy-layertest:c2 **`switch` succeeds where `upgrade` refuses.** So the rebuild-and-adopt loop the console prints is not broken by the component flip, which is the thing worth knowing before anyone "fixes" that screen. **It discards the layer, silently.** After the reboot the request is gone outright rather than left inactive, and the binary with it: 0 booted ...alloy-layertest:c2 packages: [] requested: [] 1 ...alloy-layertest:latest packages: ['alloy-demo'] requested: ['alloy-demo'] # alloy-demo alloy-demo ABSENT The rollback deployment keeps the layered state, so nothing is lost that a rollback would not restore. **On a real machine that self-heals, and it is the drain rule for free.** `etc/systemd/system/alloy-layer-components.service` gates on `ConditionPathExists=!/usr/bin/alloy`, read off the filesystem rather than off a stamp, which is exactly the case its comment says it was written for. The switch makes that condition true again, so the unit re-fires and lays the components down from the **new** image's carried repo. Confirmed by re-running the install on the switched machine with the network repo disabled: it resolves `0.0.3` out of `/usr/share/alloy-demo/rpm`, which is `c2`'s copy and not the version that was layered before. That answers, for the rebuild path, the problem recorded above as "the layer does not follow the image". On the `rpm-ostree upgrade` path a satisfied request is not re-resolved and `alloy update` has to drop and re-add. On the `bootc switch` path the drop happens on its own and the unit does the re-add, so the component tracks the image with nothing owning it. **The cost is a second reboot, and it is silent.** The user types the three commands, reboots, and lands on a machine with no console and no terminal while the unit lays them down and reboots again. That is the same silent double boot already recorded against the first-install path in GO alloy `d866e125` subtask 7 — this measurement widens it from "the first boot after an install" to "every rebuild anyone adopts", which is the more common case and the one where the user has no reason to expect it. ## The transport is not a variable: `containers-storage` behaves identically The section above switched over `--transport registry`. The update channel Alloy actually ships (wiki `alloy-update-channel`) has the machine build its own image, so the ref lives in that machine's local store and the verb is `--transport containers-storage`. Measured 2026-09-07 rather than assumed, because the transport is the one variable that design changes. Same versions, same `c1` -> `c2` shape. `c2` was copied into the **guest's** containers-storage with skopeo first, which is what a local rebuild leaves behind: # skopeo copy --src-tls-verify=false \ docker://10.0.2.2:5000/alloy-layertest:c2 \ containers-storage:localhost/alloy-layertest:c2 # bootc upgrade error: Upgrading: Deployment contains local rpm-ostree modifications; cannot upgrade via bootc. # bootc switch --transport containers-storage localhost/alloy-layertest:c2 layers already present: 69; layers needed: 4 (6.1 kB) Deploying...done (2 seconds) Queued for next boot: ostree-unverified-image:containers-storage:localhost/alloy-layertest:c2 Every result matches the registry leg: - **`switch` succeeds where `upgrade` refuses**, on the same layered deployment. - **The staged deployment reads `Diff: 1 removed`** before the reboot, so the discard is visible without waiting for it. - **The layer is gone after the reboot**, and the rollback keeps it: 0 booted ostree-unverified-image:containers-storage:...:c2 (no packages, no requested-packages) 1 ostree-unverified-registry:...:latest packages: ['alloy-demo'] requested: ['alloy-demo'] # alloy-demo alloy-demo ABSENT - **The re-layer resolves out of the new image's carried repo.** With the network repo still `enabled=0`, `rpm-ostree install -y --idempotent alloy-demo` on the switched machine adds `alloy-demo-0.0.3`, which is `c2`'s copy — not the `0.0.2` that had been layered. So the component tracks the image on this path too, and on a real machine `alloy-layer-components.service` does that re-add unprompted. **Cheaper than the registry leg, and worth knowing.** `layers needed: 4 (6.1 kB)` against 69 already present: adopting a locally built image moves almost nothing, because the ostree content store already holds what the two images share. The rebuild costs build time; the adopt costs seconds. ### One harness gotcha found doing it **`rpm-ostree install` over a non-tty ssh does not exit.** The transaction completes — the journal shows `Created new deployment` and the deployment appears with its `LayeredPackages` — and the CLI client stays connected, so `sshx 'rpm-ostree install ...'` hangs forever after the work is done. It looks exactly like a slow depsolve. `rpm-ostree status` from a second session says `State: busy` with the transaction still named, which is what distinguishes it from a real stall: read the journal, and if the deployment is there, kill the client. Anything scripting this wants `ssh -tt` or a `--` wrapper rather than a longer timeout. ## How an installed machine gets the components **They travel on the ISO**, not over the network. That is the only shape that survives an offline install, and it keeps the no-phone-home position intact for a machine whose owner has not consented to anything yet. The mechanism is the carry shape above: the repo rides inside the image, so the ISO carries it because the ISO is the image, and the installed machine has it for the same reason. No copy step in the installer, and no `/var` state to go missing. **Layering happens at first boot, not during the install.** `installtime.sh` measured the alternative and it is unproven: `rpm-ostree install` is a D-Bus client and `--sysroot` does not make it standalone, so against an unbooted target it failed three ways in a row (no bus, then a bus without activation, then a hand-started daemon that never registered the object). The first-boot shape needs no new mechanism at all, since it is the ordinary layering the cases above already measure. The difference between them is one reboot. That is not a proof that install-time layering cannot work. The failures are a container with no systemd fighting D-Bus activation, and the real installer runs inside the live ISO, which has systemd and a bus. The open part is whether rpm-ostreed will operate on a sysroot other than its own. `installtime.sh` records where to pick that up if the extra reboot ever becomes worth removing. ## Use Needs `qemu-system-x86_64` with KVM, OVMF, podman, and about 25 GB free. No swtpm and no OVMF secure-boot variables, unlike vmtest: nothing here installs to an encrypted disk. ./build.sh # three RPMs, the repo, and six base images ./serve.sh up # RPM repo on 8080, OCI registry on 5000 ./install-disk.sh n1 # install a variant to state/disk.raw ./vm.sh & # boot it, ssh lands on 2223 ./sshx 'rpm-ostree install alloy-demo-0.0.2-1.fc43.x86_64' ./sshx 'systemctl reboot' ./case.sh n3 # move the base, reboot, read the state back ./case.sh n2 --bootc # the refusal, on purpose ./case.sh n4 --drain # the wedge, with our layers dropped first ./serve.sh down # and stop the servers The full sequence that produced the table is `n1`, layer the hotfix, then `case.sh n2`, `case.sh n3`, `case.sh n4`. The drain result is `n1`, layer the hotfix, then `case.sh n4 --drain` — reinstall first, because it is the same starting state as `n4` and a disk carried over from another case is not it. Everything a run writes goes to `state/`, which is gitignored. Delete it to start clean. ## Things worth knowing before changing this **Read the JSON, not the human output.** A request that never activated prints much like one that did. `readstate.py` exists to show the `packages` versus `requested-packages` split, which is the only reliable way to tell an applied hotfix from a no-op. It is the same split `crates/alloy/src/pkg.rs:1099-1160` already parses, which is what the console would report to a user. **Reinstall between shapes, and let `install-disk.sh` delete the firmware variables.** Keeping `OVMF_VARS.fd` across a reinstall boots the new disk against the old boot entries and lands at a `grub>` prompt. **`--target-imgref` at install time is what makes upgrades possible.** Without it the installed system points at a `localhost/` reference that resolves to nothing inside the guest. `build-iso.sh --update-target` does the same job for the ISO. **The base image is deliberately not Alloy.** A run costs a 200 MB build rather than a 5 GB one, and everything measured here is rpm-ostree and bootc behaviour, which does not depend on what else the image carries. If a result ever looks like it might turn on Alloy's own content, that is the point to rebuild this against `localhost/alloy:local` rather than to argue about it. **The registry is insecure and the repo is unsigned.** Signing is its own subtask. Adding it here would add ways for a run to fail that have nothing to do with what is being measured.