Skip to main content

max / alloy

Measure the way out of the hotfix wedge, and encode it in the harness GO d866e125. The first run left the RPM route with one unanswered failure, and it was the normal life of a hotfix rather than an edge case: ship 0.0.2 out of band, fold it into the next image at 0.0.3, and every machine that took the hotfix stops updating forever with `cannot install both 0.0.2 from alloy-demo and 0.0.3 from @System`. The rule that answers it is that our layers are disposable. Never carry one across an upgrade: drop it first, upgrade, and re-apply afterwards only if the base still lacks the fix. A layer and a base copy are then never in the same transaction, so the conflict cannot arise. Measured, from a machine on n1 holding a layered 0.0.2, against a base carrying 0.0.3. It holds, and it is cheaper than expected: the uninstall and the upgrade each stage a deployment and the second composes onto the first, so it is one reboot rather than two, and the rollback deployment keeps the layered state. `bootc upgrade` consents again afterwards, so a machine that took a hotfix is not stranded on `rpm-ostree upgrade` for the rest of its life either. One sharp edge, and it cost a run to find. A request is recorded under the string that was typed to install it, so a package layered by full NEVRA cannot be removed by its bare name: `rpm-ostree uninstall alloy-demo` answers "Package/capability 'alloy-demo' is not currently requested" while alloy-demo-0.0.2-1.fc43.x86_64 sits in requested-packages. That reads as nothing-to-do rather than as a failure, so a drain written against the package name silently does not drain and the machine wedges exactly as if the rule had never been applied. `readstate.py --requested <prefix>` is the lookup that avoids it, and `case.sh --drain` uses it. Enumerated rather than `rpm-ostree reset`, which would work here and is wrong as a model: reset drops every layered package including ones the user layered themselves, and those are not ours to remove. Production has to be precise, so the harness is. `case.sh` also reports whether the machine can still take the next update, both verbs. A wedge does not show up in the deployment state — it reports a perfectly ordinary deployment and only announces itself at the next upgrade.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-14 22:46 UTC
Signed with PGP, not checked
Commit: d19672101687ec055c78e97a068fefdea25de8f6
Parent: 9458942
3 files changed, +120 insertions, -3 deletions
M build/layertest/README.md +45 -1
@@ -69,6 +69,47 @@
69 69 Recovery is `rpm-ostree uninstall` and then upgrade, after which `bootc
70 70 upgrade` works again, but a user has no way to know that is what happened.
71 71
72 + ## The wedge is avoidable, 2026-08-14
73 +
74 + Measured after the above, on the same versions. The rule is that **our layers
75 + are disposable**: never carried across an upgrade, dropped before it, and
76 + re-applied afterwards only if the base still lacks the fix. If no layer and no
77 + base copy are ever in the same transaction, the depsolve conflict cannot arise.
78 +
79 + It holds. From a machine on `n1` holding a layered `alloy-demo-0.0.2`, against
80 + `n4` (base carrying 0.0.3), `case.sh n4 --drain`:
81 +
82 + Removed: alloy-demo-0.0.2-1.fc43.x86_64
83 + Upgraded: alloy-demo 0.0.2-1.fc43 -> 0.0.3-1.fc43
84 +
85 + base-mark: n4 binary: alloy-demo 0.0.3 no layers
86 +
87 + **One reboot, not two.** The uninstall and the upgrade each stage a deployment
88 + and the second composes onto the first, so both land in one boot. The rollback
89 + deployment still holds the layered state, which is what a rollback is for.
90 +
91 + **`bootc upgrade` consents again.** With no local modifications left, the
92 + machine is back on the bootc path rather than stranded on `rpm-ostree upgrade`
93 + forever, which was the other cost the first run turned up.
94 +
95 + **The sharp edge, and it cost a run to find.** A request is recorded under
96 + whatever string was typed to install it, so a package layered by full NEVRA
97 + cannot be removed by its bare name:
98 +
99 + # rpm-ostree uninstall alloy-demo
100 + error: Package/capability 'alloy-demo' is not currently requested
101 +
102 + with `alloy-demo-0.0.2-1.fc43.x86_64` sitting in `requested-packages` the whole
103 + time. That reads as "there was nothing to do" rather than as a failure, so a
104 + drain written against the package name silently does not drain and the machine
105 + wedges exactly as if the rule had never been applied. Anything implementing
106 + this has to read `requested-packages` and uninstall the exact strings it finds;
107 + `readstate.py --requested <prefix>` is that lookup, and `--drain` uses it.
108 +
109 + Not `rpm-ostree reset`, though bootc's own error message suggests it and it
110 + would work here. Reset drops every layered package including ones the user
111 + layered themselves, and those are not ours to remove.
112 +
72 113 ## Use
73 114
74 115 Needs `qemu-system-x86_64` with KVM, OVMF, podman, and about 25 GB free. No
@@ -83,10 +124,13 @@
83 124 ./sshx 'systemctl reboot'
84 125 ./case.sh n3 # move the base, reboot, read the state back
85 126 ./case.sh n2 --bootc # the refusal, on purpose
127 + ./case.sh n4 --drain # the wedge, with our layers dropped first
86 128 ./serve.sh down # and stop the servers
87 129
88 130 The full sequence that produced the table is `n1`, layer the hotfix, then
89 - `case.sh n2`, `case.sh n3`, `case.sh n4`.
131 + `case.sh n2`, `case.sh n3`, `case.sh n4`. The drain result is `n1`, layer the
132 + hotfix, then `case.sh n4 --drain` — reinstall first, because it is the same
133 + starting state as `n4` and a disk carried over from another case is not it.
90 134
91 135 Everything a run writes goes to `state/`, which is gitignored. Delete it to
92 136 start clean.
M build/layertest/case.sh +55 -2
@@ -5,6 +5,20 @@
5 5 #
6 6 # case.sh n2 upgrade with rpm-ostree, reboot, read the state back
7 7 # case.sh n2 --bootc use bootc upgrade instead, which is a separate answer
8 + # case.sh n4 --drain drop our layers first, then upgrade
9 + #
10 + # --drain is the candidate answer to the wedge the first run measured: a base
11 + # that moves ahead of a layered hotfix stops the machine updating, forever,
12 + # and that is the normal life of a hotfix rather than an edge case. The rule
13 + # it tests is that our own layers are disposable — never carried across an
14 + # upgrade, dropped before it and re-applied after if the base still needs
15 + # them. If that holds, the depsolve conflict cannot arise at all, because
16 + # there is never a layer and a base copy in the same transaction.
17 + #
18 + # Chained into one transaction on purpose. rpm-ostree stages a deployment per
19 + # command, and what matters is whether a user pays one reboot or two, so the
20 + # uninstall and the upgrade go in a single invocation and the deployment count
21 + # is part of the result.
8 22 #
9 23 # Retagging is how the base moves. The installed system points at :latest in
10 24 # the host registry, so pushing a different variant to that tag is the whole
@@ -22,13 +36,43 @@
22 36 . "$HERE/../privilege.sh"
23 37
24 38 MARK="${1:-}"
25 - [ -n "$MARK" ] || die "usage: case.sh <mark> [--bootc]"
39 + [ -n "$MARK" ] || die "usage: case.sh <mark> [--bootc|--drain]"
26 40 VERB="rpm-ostree upgrade"
27 - [ "${2:-}" = "--bootc" ] && VERB="bootc upgrade"
41 + DRAIN=no
42 + case "${2:-}" in
43 + --bootc) VERB="bootc upgrade" ;;
44 + --drain) DRAIN=yes ;;
45 + "") ;;
46 + *) die "unknown option ${2}" ;;
47 + esac
28 48
29 49 privc podman push --tls-verify=false "$IMAGE:$MARK" \
30 50 "127.0.0.1:$REGISTRY_PORT/alloy-layertest:latest" >/dev/null 2>&1
31 51
52 + # Drop our own layers before the upgrade rather than after, and by the exact
53 + # string the request was recorded under. A bare package name does not match a
54 + # request made by NEVRA, and rpm-ostree says so in a way that reads as "there
55 + # was nothing to do" rather than as a failure — which is how a drain silently
56 + # does not drain and the machine wedges anyway.
57 + #
58 + # Enumerated rather than `rpm-ostree reset`, which would work here and is
59 + # wrong as a model: reset drops every layered package including ones the user
60 + # layered themselves, and those are not ours to remove. Production has to be
61 + # this precise, so the harness is too.
62 + if [ "$DRAIN" = yes ]; then
63 + say "=== drain: dropping our layers before the upgrade ==="
64 + ours=$("$HERE/sshx" 'rpm-ostree status --json' \
65 + | python3 "$HERE/readstate.py" --requested alloy-demo)
66 + if [ -z "$ours" ]; then
67 + say "nothing of ours is layered"
68 + else
69 + for package in $ours; do
70 + say "uninstalling $package"
71 + "$HERE/sshx" "rpm-ostree uninstall $package" 2>&1 | tail -3
72 + done
73 + fi
74 + fi
75 +
32 76 say "=== $VERB onto $MARK ==="
33 77 # Not fatal. A refusal is a result, and the whole point of --bootc is to see
34 78 # one, so the script has to survive the command it is measuring.
@@ -42,3 +86,12 @@
42 86 echo -n "binary: "; alloy-demo 2>&1 || echo "(absent)"; \
43 87 echo -n "rpm -q: "; rpm -q alloy-demo 2>&1' || die "guest did not come back"
44 88 "$HERE/sshx" 'rpm-ostree status --json' | python3 "$HERE/readstate.py"
89 +
90 + # Whether the machine can still take the next one. A wedge does not show up in
91 + # the state above — it reports a perfectly ordinary deployment — and only
92 + # announces itself the next time somebody tries to update. Both verbs, because
93 + # the whole point of dropping the layer is to hand `bootc upgrade` back a
94 + # deployment it will consent to work on.
95 + say "--- can it still update? ---"
96 + "$HERE/sshx" 'rpm-ostree upgrade --check 2>&1 | tail -3' || true
97 + "$HERE/sshx" 'bootc upgrade --check 2>&1 | tail -3' || true
M build/layertest/readstate.py +20
@@ -8,6 +8,15 @@
8 8 # gets mistaken for an applied hotfix.
9 9 #
10 10 # ./sshx 'rpm-ostree status --json' | python3 readstate.py
11 + #
12 + # With --requested PREFIX it prints nothing but the requested-packages of the
13 + # booted deployment whose name starts with PREFIX, one per line, for a caller
14 + # that means to uninstall them. The exact strings matter: a request is recorded
15 + # under whatever was typed to install it, so a package installed by full NEVRA
16 + # cannot be removed by its bare name — `rpm-ostree uninstall alloy-demo` on a
17 + # machine holding `alloy-demo-0.0.2-1.fc43.x86_64` answers "not currently
18 + # requested" and leaves it in place. Measured 2026-08-14, and it is the whole
19 + # reason this mode exists rather than the caller writing a package name.
11 20
12 21 import json
13 22 import sys
@@ -20,6 +29,17 @@
20 29 )
21 30
22 31 data = json.load(sys.stdin)
32 +
33 + if len(sys.argv) > 2 and sys.argv[1] == "--requested":
34 + prefix = sys.argv[2]
35 + for deployment in data["deployments"]:
36 + if not deployment.get("booted"):
37 + continue
38 + for package in deployment.get("requested-packages", []):
39 + if package.startswith(prefix):
40 + print(package)
41 + sys.exit(0)
42 +
23 43 for index, deployment in enumerate(data["deployments"]):
24 44 print("--- deployment %d booted=%s staged=%s" % (
25 45 index, deployment.get("booted"), deployment.get("staged")))