| 1 |
# layertest — what a hotfix is allowed to do to an installed machine |
| 2 |
|
| 3 |
Alloy is distributed as a builder rather than as an image, so there is no way |
| 4 |
to hand a running machine a fix short of "rebuild the ISO and write a drive". |
| 5 |
The proposed answer is a signed RPM repo carrying only our own components, |
| 6 |
layered with `rpm-ostree` (GoingsOn task `d866e125`). That answer rests on one |
| 7 |
assumption: that a package layered today is still there, and still correct, |
| 8 |
after the base moves tomorrow. |
| 9 |
|
| 10 |
This harness measures that assumption instead of trusting it. It is not a test |
| 11 |
suite and nothing runs it automatically. It exists to be re-run when rpm-ostree |
| 12 |
or bootc moves, because every result below is a behaviour of those two and not |
| 13 |
of anything in this repo. |
| 14 |
|
| 15 |
## What it found |
| 16 |
|
| 17 |
Measured on rpm-ostree 2026.1, bootc 1.16.3, libostree 2026.2, against a |
| 18 |
`quay.io/fedora/fedora-bootc:43` base. |
| 19 |
|
| 20 |
**`bootc upgrade` refuses to run on a layered deployment.** It does not discard |
| 21 |
the layer and does not keep it. It errors and does nothing: |
| 22 |
|
| 23 |
error: Upgrading: Deployment contains local rpm-ostree modifications; |
| 24 |
cannot upgrade via bootc. You can run `rpm-ostree reset` to undo the |
| 25 |
modifications. |
| 26 |
|
| 27 |
The two mechanisms are mutually exclusive. `rpm-ostree upgrade` is the verb |
| 28 |
that works on a layered system, and it pulls the same container image, so |
| 29 |
anything shipping a hotfix channel has to move machines off `bootc upgrade`. |
| 30 |
|
| 31 |
**A component already in the base cannot be replaced from a repo.** With the |
| 32 |
component installed as a base RPM at 0.0.1 and 0.0.2 in the repo: |
| 33 |
|
| 34 |
- `rpm-ostree install` fails to depsolve, `cannot install both ... from |
| 35 |
@System`. |
| 36 |
- `rpm-ostree override replace` reports `Inactive base replacements` and writes |
| 37 |
no new commit, in both of its forms. They differ only in what they leave |
| 38 |
behind, and neither difference helps: given a local RPM or a URL, the request |
| 39 |
is recorded in `requested-base-local-replacements` and survives a reboot as a |
| 40 |
silent no-op, so the machine reports a hotfix it is not running. Given |
| 41 |
`--experimental --from repo=`, nothing is recorded at all once the machine |
| 42 |
reboots, so the request disappears without ever having done anything. |
| 43 |
- `override remove` on a base package does commit, but pairing it with |
| 44 |
`--install` fails the same depsolve. |
| 45 |
|
| 46 |
Reproduce with variant `r1`. |
| 47 |
|
| 48 |
**A component that is an unowned file cannot be layered over at all.** A `COPY` |
| 49 |
line putting a binary into `/usr/bin` leaves it owned by no package, and this |
| 50 |
measurement is why Alloy does not ship one. Layering an RPM that carries the |
| 51 |
same path dies in checkout rather than in depsolve: |
| 52 |
|
| 53 |
error: Checkout alloy-demo-0.0.2: Hardlinking ... to alloy-demo: File exists |
| 54 |
|
| 55 |
Reproduce with variant `l1`. |
| 56 |
|
| 57 |
**The shape that does work, and where it breaks.** If the base does not carry |
| 58 |
the component at all, the hotfix is a plain layered package and applies |
| 59 |
cleanly. Across `rpm-ostree upgrade`: |
| 60 |
|
| 61 |
|
| 62 |
|
| 63 |
| a base still without the component (`n2`) | survives, re-applies, clean | |
| 64 |
| a base carrying it at the same version (`n3`) | goes `Inactive requests: already provided by`, drops out of `packages`, stays in `requested-packages`. Self-healing | |
| 65 |
| a base carrying a **newer** version (`n4`) | **the machine stops updating**, `cannot install both 0.0.2 from alloy-demo and 0.0.3 from @System`, and every later upgrade fails too | |
| 66 |
|
| 67 |
That last row is the normal life of a hotfix: ship it out of band, fold it into |
| 68 |
the next image at a higher version, and every machine that took it wedges. |
| 69 |
Recovery is `rpm-ostree uninstall` and then upgrade, after which `bootc |
| 70 |
upgrade` works again, but a user has no way to know that is what happened. |
| 71 |
|
| 72 |
## The wedge is avoidable |
| 73 |
|
| 74 |
Measured after the above, on the same versions. The rule is that **our layers |
| 75 |
are disposable**: never carried across an upgrade, dropped before it, and |
| 76 |
re-applied afterwards only if the base still lacks the fix. If no layer and no |
| 77 |
base copy are ever in the same transaction, the depsolve conflict cannot arise. |
| 78 |
|
| 79 |
It holds. From a machine on `n1` holding a layered `alloy-demo-0.0.2`, against |
| 80 |
`n4` (base carrying 0.0.3), `case.sh n4 --drain`: |
| 81 |
|
| 82 |
Removed: alloy-demo-0.0.2-1.fc43.x86_64 |
| 83 |
Upgraded: alloy-demo 0.0.2-1.fc43 -> 0.0.3-1.fc43 |
| 84 |
|
| 85 |
base-mark: n4 binary: alloy-demo 0.0.3 no layers |
| 86 |
|
| 87 |
**One reboot, not two.** The uninstall and the upgrade each stage a deployment |
| 88 |
and the second composes onto the first, so both land in one boot. The rollback |
| 89 |
deployment still holds the layered state, which is what a rollback is for. |
| 90 |
|
| 91 |
**`bootc upgrade` consents again.** With no local modifications left, the |
| 92 |
machine is back on the bootc path rather than stranded on `rpm-ostree upgrade` |
| 93 |
forever. |
| 94 |
|
| 95 |
**The sharp edge.** A request is recorded under |
| 96 |
whatever string was typed to install it, so a package layered by full NEVRA |
| 97 |
cannot be removed by its bare name: |
| 98 |
|
| 99 |
# rpm-ostree uninstall alloy-demo |
| 100 |
error: Package/capability 'alloy-demo' is not currently requested |
| 101 |
|
| 102 |
with `alloy-demo-0.0.2-1.fc43.x86_64` sitting in `requested-packages` the whole |
| 103 |
time. That reads as "there was nothing to do" rather than as a failure, so a |
| 104 |
drain written against the package name silently does not drain and the machine |
| 105 |
wedges exactly as if the rule had never been applied. Anything implementing |
| 106 |
this has to read `requested-packages` and uninstall the exact strings it finds; |
| 107 |
`readstate.py --requested <prefix>` is that lookup, and `--drain` uses it. |
| 108 |
|
| 109 |
Not `rpm-ostree reset`, though bootc's own error message suggests it and it |
| 110 |
would work here. Reset drops every layered package including ones the user |
| 111 |
layered themselves, and those are not ours to remove. |
| 112 |
|
| 113 |
## The carry shape, and it is the one Alloy takes |
| 114 |
|
| 115 |
The image carries our RPMs **as files**, in a `file://` repo under |
| 116 |
`/usr/share/alloy/rpm`, and does not install them. The component is still |
| 117 |
absent from `@System`, so it layers exactly as in the `none` cases, but the |
| 118 |
package it layers from travels inside the image. Variants `c1` (carries 0.0.2) |
| 119 |
and `c2` (carries 0.0.3). |
| 120 |
|
| 121 |
Nothing has to be copied anywhere at install time. The ISO is the image, so the |
| 122 |
ISO carries the repo; the installed machine is the image, so it carries the |
| 123 |
repo too. That is the whole delivery mechanism. |
| 124 |
|
| 125 |
**Layering works with no network.** With the network repo `enabled=0`, from a |
| 126 |
`c1` machine: |
| 127 |
|
| 128 |
# rpm-ostree install -y --idempotent alloy-demo |
| 129 |
Added: alloy-demo-0.0.2-1.fc43.x86_64 |
| 130 |
|
| 131 |
Requested under the **bare name**, because that is what was typed. Worth |
| 132 |
noticing next to the NEVRA trap recorded below: install by name and the drain |
| 133 |
can uninstall by name. |
| 134 |
|
| 135 |
**A base move does not wedge it.** `c1` to `c2`, base carrying 0.0.3 against a |
| 136 |
layer at 0.0.2, no drain, plain `rpm-ostree upgrade`: it succeeds. This is the |
| 137 |
whole reason to prefer this shape over shipping the component as a base RPM. |
| 138 |
`@System` never holds the component, so the conflict that permanently wedges |
| 139 |
the `n4` case has nothing to arise between. |
| 140 |
|
| 141 |
**But the layer does not follow the image.** After that upgrade the machine is |
| 142 |
on base `c2` and still running `alloy-demo 0.0.2`, while the base beside it |
| 143 |
carries 0.0.3. rpm-ostree does not re-resolve a satisfied request just because |
| 144 |
a repo behind it moved. Shipping a new console inside a new image therefore |
| 145 |
reaches nobody on its own, which is the opposite of what it looks like. |
| 146 |
|
| 147 |
**What advances it is a drop and reinstall**, in one boot: |
| 148 |
|
| 149 |
# rpm-ostree uninstall alloy-demo && rpm-ostree install -y alloy-demo |
| 150 |
Upgraded: alloy-demo 0.0.2-1.fc43 -> 0.0.3-1.fc43 |
| 151 |
|
| 152 |
The drain rule holds here for a second reason: it is the only thing that makes |
| 153 |
a component track the image it shipped with. `alloy update` owns it. |
| 154 |
|
| 155 |
**`bootc upgrade` refuses permanently, not transiently.** An Alloy machine |
| 156 |
always carries a layer, so it always has "local rpm-ostree modifications". |
| 157 |
`alloy update` owns `rpm-ostree upgrade` for the life of the machine, and |
| 158 |
moving users off `bootc upgrade` is a permanent condition rather than a step in |
| 159 |
a migration. |
| 160 |
|
| 161 |
## `bootc switch` is not `bootc upgrade` |
| 162 |
|
| 163 |
The paragraph above is about `upgrade`, and reads as though the whole bootc |
| 164 |
path were closed to a layered machine. It is not, and the difference matters |
| 165 |
because **`bootc switch` is the command Alloy actually prints**: it is the |
| 166 |
middle line of `Staleness::REMEDY` in `crates/alloy/src/stale.rs` and the |
| 167 |
adoption step in docs/IMAGE.md. |
| 168 |
|
| 169 |
Measured on the same versions, from a `c1` machine holding a layered |
| 170 |
`alloy-demo`, switching to `c2`: |
| 171 |
|
| 172 |
# bootc upgrade |
| 173 |
error: Upgrading: Deployment contains local rpm-ostree modifications; |
| 174 |
cannot upgrade via bootc. |
| 175 |
|
| 176 |
# bootc switch --transport registry 10.0.2.2:5000/alloy-layertest:c2 |
| 177 |
Queued for next boot: 10.0.2.2:5000/alloy-layertest:c2 |
| 178 |
|
| 179 |
**`switch` succeeds where `upgrade` refuses.** So the rebuild-and-adopt loop |
| 180 |
the console prints is not broken by the component flip, which is the thing |
| 181 |
worth knowing before anyone "fixes" that screen. |
| 182 |
|
| 183 |
**It discards the layer, silently.** After the reboot the request is gone |
| 184 |
outright rather than left inactive, and the binary with it: |
| 185 |
|
| 186 |
0 booted ...alloy-layertest:c2 |
| 187 |
packages: [] requested: [] |
| 188 |
1 ...alloy-layertest:latest |
| 189 |
packages: ['alloy-demo'] requested: ['alloy-demo'] |
| 190 |
|
| 191 |
# alloy-demo |
| 192 |
alloy-demo ABSENT |
| 193 |
|
| 194 |
The rollback deployment keeps the layered state, so nothing is lost that a |
| 195 |
rollback would not restore. |
| 196 |
|
| 197 |
**On a real machine that self-heals, and it is the drain rule for free.** |
| 198 |
`etc/systemd/system/alloy-layer-components.service` gates on |
| 199 |
`ConditionPathExists=!/usr/bin/alloy`, read off the filesystem rather than off |
| 200 |
a stamp, which is exactly the case its comment says it was written for. The |
| 201 |
switch makes that condition true again, so the unit re-fires and lays the |
| 202 |
components down from the **new** image's carried repo. Confirmed by re-running |
| 203 |
the install on the switched machine with the network repo disabled: it |
| 204 |
resolves `0.0.3` out of `/usr/share/alloy-demo/rpm`, which is `c2`'s copy and |
| 205 |
not the version that was layered before. |
| 206 |
|
| 207 |
That answers, for the rebuild path, the problem recorded above as "the layer |
| 208 |
does not follow the image". On the `rpm-ostree upgrade` path a satisfied |
| 209 |
request is not re-resolved and `alloy update` has to drop and re-add. On the |
| 210 |
`bootc switch` path the drop happens on its own and the unit does the re-add, |
| 211 |
so the component tracks the image with nothing owning it. |
| 212 |
|
| 213 |
**The cost is a second reboot, and it is silent.** The user types the three |
| 214 |
commands, reboots, and lands on a machine with no console and no terminal |
| 215 |
while the unit lays them down and reboots again. That is the same silent |
| 216 |
double boot already recorded against the first-install path in GO alloy |
| 217 |
`d866e125` subtask 7 — this measurement widens it from "the first boot after |
| 218 |
an install" to "every rebuild anyone adopts", which is the more common case |
| 219 |
and the one where the user has no reason to expect it. |
| 220 |
|
| 221 |
## The transport is not a variable: `containers-storage` behaves identically |
| 222 |
|
| 223 |
The section above switched over `--transport registry`. The update channel |
| 224 |
Alloy actually ships (wiki `alloy-update-channel`) has the machine build its |
| 225 |
own image, so the ref lives in that machine's local store and the verb is |
| 226 |
`--transport containers-storage`. Measured 2026-09-07 rather than assumed, |
| 227 |
because the transport is the one variable that design changes. |
| 228 |
|
| 229 |
Same versions, same `c1` -> `c2` shape. `c2` was copied into the **guest's** |
| 230 |
containers-storage with skopeo first, which is what a local rebuild leaves |
| 231 |
behind: |
| 232 |
|
| 233 |
# skopeo copy --src-tls-verify=false \ |
| 234 |
docker://10.0.2.2:5000/alloy-layertest:c2 \ |
| 235 |
containers-storage:localhost/alloy-layertest:c2 |
| 236 |
|
| 237 |
# bootc upgrade |
| 238 |
error: Upgrading: Deployment contains local rpm-ostree modifications; |
| 239 |
cannot upgrade via bootc. |
| 240 |
|
| 241 |
# bootc switch --transport containers-storage localhost/alloy-layertest:c2 |
| 242 |
layers already present: 69; layers needed: 4 (6.1 kB) |
| 243 |
Deploying...done (2 seconds) |
| 244 |
Queued for next boot: |
| 245 |
ostree-unverified-image:containers-storage:localhost/alloy-layertest:c2 |
| 246 |
|
| 247 |
Every result matches the registry leg: |
| 248 |
|
| 249 |
- **`switch` succeeds where `upgrade` refuses**, on the same layered deployment. |
| 250 |
- **The staged deployment reads `Diff: 1 removed`** before the reboot, so the |
| 251 |
discard is visible without waiting for it. |
| 252 |
- **The layer is gone after the reboot**, and the rollback keeps it: |
| 253 |
|
| 254 |
0 booted ostree-unverified-image:containers-storage:...:c2 |
| 255 |
(no packages, no requested-packages) |
| 256 |
1 ostree-unverified-registry:...:latest |
| 257 |
packages: ['alloy-demo'] requested: ['alloy-demo'] |
| 258 |
|
| 259 |
# alloy-demo |
| 260 |
alloy-demo ABSENT |
| 261 |
|
| 262 |
- **The re-layer resolves out of the new image's carried repo.** With the |
| 263 |
network repo still `enabled=0`, `rpm-ostree install -y --idempotent |
| 264 |
alloy-demo` on the switched machine adds `alloy-demo-0.0.3`, which is `c2`'s |
| 265 |
copy — not the `0.0.2` that had been layered. So the component tracks the |
| 266 |
image on this path too, and on a real machine |
| 267 |
`alloy-layer-components.service` does that re-add unprompted. |
| 268 |
|
| 269 |
**Cheaper than the registry leg, and worth knowing.** `layers needed: 4 |
| 270 |
(6.1 kB)` against 69 already present: adopting a locally built image moves |
| 271 |
almost nothing, because the ostree content store already holds what the two |
| 272 |
images share. The rebuild costs build time; the adopt costs seconds. |
| 273 |
|
| 274 |
### One harness gotcha found doing it |
| 275 |
|
| 276 |
**`rpm-ostree install` over a non-tty ssh does not exit.** The transaction |
| 277 |
completes — the journal shows `Created new deployment` and the deployment |
| 278 |
appears with its `LayeredPackages` — and the CLI client stays connected, so |
| 279 |
`sshx 'rpm-ostree install ...'` hangs forever after the work is done. It looks |
| 280 |
exactly like a slow depsolve. `rpm-ostree status` from a second session says |
| 281 |
`State: busy` with the transaction still named, which is what distinguishes it |
| 282 |
from a real stall: read the journal, and if the deployment is there, kill the |
| 283 |
client. Anything scripting this wants `ssh -tt` or a `--` wrapper rather than |
| 284 |
a longer timeout. |
| 285 |
|
| 286 |
## How an installed machine gets the components |
| 287 |
|
| 288 |
**They travel on the ISO**, not over the network. That is the only shape that survives an offline install, and it keeps the no-phone-home |
| 289 |
position intact for a machine whose owner has not consented to anything yet. |
| 290 |
|
| 291 |
The mechanism is the carry shape above: the repo rides inside the image, so the |
| 292 |
ISO carries it because the ISO is the image, and the installed machine has it |
| 293 |
for the same reason. No copy step in the installer, and no `/var` state to go |
| 294 |
missing. |
| 295 |
|
| 296 |
**Layering happens at first boot, not during the install.** `installtime.sh` |
| 297 |
measured the alternative and it is unproven: `rpm-ostree install` is a D-Bus |
| 298 |
client and `--sysroot` does not make it standalone, so against an unbooted |
| 299 |
target it failed three ways in a row (no bus, then a bus without activation, |
| 300 |
then a hand-started daemon that never registered the object). The first-boot |
| 301 |
shape needs no new mechanism at all, since it is the ordinary layering the |
| 302 |
cases above already measure. The difference between them is one reboot. |
| 303 |
|
| 304 |
That is not a proof that install-time layering cannot work. The failures are a |
| 305 |
container with no systemd fighting D-Bus activation, and the real installer |
| 306 |
runs inside the live ISO, which has systemd and a bus. The open part is whether |
| 307 |
rpm-ostreed will operate on a sysroot other than its own. `installtime.sh` |
| 308 |
records where to pick that up if the extra reboot ever becomes worth removing. |
| 309 |
|
| 310 |
## Use |
| 311 |
|
| 312 |
Needs `qemu-system-x86_64` with KVM, OVMF, podman, and about 25 GB free. No |
| 313 |
swtpm and no OVMF secure-boot variables, unlike vmtest: nothing here installs |
| 314 |
to an encrypted disk. |
| 315 |
|
| 316 |
./build.sh # three RPMs, the repo, and six base images |
| 317 |
./serve.sh up # RPM repo on 8080, OCI registry on 5000 |
| 318 |
./install-disk.sh n1 # install a variant to state/disk.raw |
| 319 |
./vm.sh & # boot it, ssh lands on 2223 |
| 320 |
./sshx 'rpm-ostree install alloy-demo-0.0.2-1.fc43.x86_64' |
| 321 |
./sshx 'systemctl reboot' |
| 322 |
./case.sh n3 # move the base, reboot, read the state back |
| 323 |
./case.sh n2 --bootc # the refusal, on purpose |
| 324 |
./case.sh n4 --drain # the wedge, with our layers dropped first |
| 325 |
./serve.sh down # and stop the servers |
| 326 |
|
| 327 |
The full sequence that produced the table is `n1`, layer the hotfix, then |
| 328 |
`case.sh n2`, `case.sh n3`, `case.sh n4`. The drain result is `n1`, layer the |
| 329 |
hotfix, then `case.sh n4 --drain` — reinstall first, because it is the same |
| 330 |
starting state as `n4` and a disk carried over from another case is not it. |
| 331 |
|
| 332 |
Everything a run writes goes to `state/`, which is gitignored. Delete it to |
| 333 |
start clean. |
| 334 |
|
| 335 |
## Things worth knowing before changing this |
| 336 |
|
| 337 |
**Read the JSON, not the human output.** A request that never activated prints |
| 338 |
much like one that did. `readstate.py` exists to show the `packages` versus |
| 339 |
`requested-packages` split, which is the only reliable way to tell an applied |
| 340 |
hotfix from a no-op. It is the same split `crates/alloy/src/pkg.rs:1099-1160` |
| 341 |
already parses, which is what the console would report to a user. |
| 342 |
|
| 343 |
**Reinstall between shapes, and let `install-disk.sh` delete the firmware |
| 344 |
variables.** Keeping `OVMF_VARS.fd` across a reinstall boots the new disk |
| 345 |
against the old boot entries and lands at a `grub>` prompt. |
| 346 |
|
| 347 |
**`--target-imgref` at install time is what makes upgrades possible.** Without |
| 348 |
it the installed system points at a `localhost/` reference that resolves to |
| 349 |
nothing inside the guest. `build-iso.sh --update-target` does the same job for |
| 350 |
the ISO. |
| 351 |
|
| 352 |
**The base image is deliberately not Alloy.** A run costs a 200 MB build rather |
| 353 |
than a 5 GB one, and everything measured here is rpm-ostree and bootc |
| 354 |
behaviour, which does not depend on what else the image carries. If a result |
| 355 |
ever looks like it might turn on Alloy's own content, that is the point to |
| 356 |
rebuild this against `localhost/alloy:local` rather than to argue about it. |
| 357 |
|
| 358 |
**The registry is insecure and the repo is unsigned.** Signing is its own |
| 359 |
subtask. Adding it here would add ways for a run to fail that have nothing to |
| 360 |
do with what is being measured. |
| 361 |
|