Skip to main content

max / alloy

Gate the SHOP_REV bump on a build that actually happened `ARG SHOP_REV` is a one-line edit that looks like it cannot fail. shop links its system libraries rather than dlopening them, so a revision can arrive needing a -devel package the rust-build stage does not install: shop@53551eb added fontconfig, alloy@0eb6465 moved the pin onto it, both profiles died at `cargo install shop`, and the commit reached astra, srht and mnw before anyone built it. `cargo test --workspace` cannot see any of that, because shop is fetched and built inside the container. `build/check-rust-stage.sh` was already the answer and was already documented as the thing to run before pushing a bump. On 2026-08-09 nobody ran it. So the instruction becomes a gate rather than a second instruction: a clean run records the revision in `build/rust-stage-verified`, and `crates/alloy/tests/shop_rev_verified.rs` fails while the Containerfile names a different one. An unchecked bump is refused by `cargo test` in a second rather than by an hour of podman on someone else's machine. Not the other shape, and the script's own header says why: a static check that reads shop's manifest and asserts every dependency has its -devel package needs a crate-to-package table that is not mechanical, and passes by default for anything the table has not learned. The build stays the evidence; the stamp is only the record that the evidence exists. Written on success only, by the script only, and the content is the revision alone so a run at an already-recorded revision rewrites the file byte-for-byte and leaves the nightly sweep's worktree clean. Also: a dead base pin was classified as exit 1, "the stage does not build at SHOP_REV af8c79c2" -- a claim about shop's revision on evidence that never reached shop. `manifest unknown` and `creating build container` join the network patterns, so it exits 3 and points at refresh-base-digests.sh, whose finding it is. Two self-test fixtures cover it, from the log that produced it. Seeded by a real run: the rust-build stage built end to end on astra at SHOP_REV af8c79c2, on the refreshed bases, and wrote the stamp itself.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-22 00:12 UTC
Signed with PGP, not checked
Commit: 989af718986479ca36e0c163bf147ff547c6b5f7
Parent: b1beb31
4 files changed, +194 insertions, -9 deletions
M Containerfile +10 -3
@@ -125,9 +125,16 @@
125 125 # Bump this deliberately, and run `build/check-rust-stage.sh` before pushing
126 126 # the bump: shop links its system libraries rather than dlopening them, so a
127 127 # revision can arrive needing a -devel package the dnf line above does not
128 - # install, and this file is where that has to be fixed. The astra sweep runs
129 - # the same script nightly (`rust-stage`), which is the backstop for when
130 - # nobody does; on 2026-08-09 nobody did and the break reached three remotes.
128 + # install, and this file is where that has to be fixed. On 2026-08-09 nobody
129 + # ran it and the break reached three remotes.
130 + #
131 + # That sentence used to end there, as an instruction. It is a gate now: a clean
132 + # run records the revision in `build/rust-stage-verified`, and
133 + # `crates/alloy/tests/shop_rev_verified.rs` fails while this line names a
134 + # different one, so an unchecked bump is refused by `cargo test` rather than by
135 + # an hour of podman. The astra sweep still runs the script nightly
136 + # (`rust-stage`), which is the backstop for a pinned revision that stops
137 + # building without anyone editing this file.
131 138 #
132 139 # `cargo install` rather than a second COPY-and-build stage: shop is a separate
133 140 # repo with its own workspace, so there is no local tree to copy and no
M build/check-rust-stage.sh +58 -6
@@ -42,8 +42,14 @@
42 42 # fails invisibly: a wrong verdict still looks like a verdict. Run it after
43 43 # touching any of the three patterns.
44 44 #
45 - # Run it after moving SHOP_REV, before pushing. The astra sweep runs it nightly
46 - # (`rust-stage` in sweep.toml), which is the backstop for when nobody does.
45 + # Run it after moving SHOP_REV, before pushing. That used to be an instruction
46 + # and nothing else, which is how alloy@0eb6465 shipped a rev that could not
47 + # build. A clean run now records the revision in `build/rust-stage-verified`,
48 + # and `crates/alloy/tests/shop_rev_verified.rs` fails while the Containerfile
49 + # names a different one — so the bump is refused by `cargo test` in a second,
50 + # rather than by an hour of podman on someone else's machine. The astra sweep
51 + # runs this script nightly (`rust-stage` in sweep.toml), which stays the
52 + # backstop for a rev that stops building without anyone touching the pin.
47 53 #
48 54 # Exit codes follow refresh-base-digests.sh, for the same reason:
49 55 #
@@ -51,8 +57,11 @@
51 57 # 1 the stage does not build. The image is unbuildable and, since the
52 58 # install path is "build it yourself" (docs/IMAGE.md), so is Alloy.
53 59 # 3 something about the run, not about the image: no podman, no network, a
54 - # registry or a git remote nobody could reach. Prints `error:`, which the
55 - # sweep turns into an error cell rather than a finding.
60 + # registry or a git remote nobody could reach, or a pinned base digest that
61 + # is gone from the registry. Prints `error:`, which the sweep turns into an
62 + # error cell rather than a finding. The last of those is a real defect, but
63 + # it belongs to `base-pins`, which names it DEAD and exits nonzero; here it
64 + # is only a build that never started.
56 65 #
57 66 # Rootless on purpose, unlike build-image.sh, which needs rootful podman for
58 67 # the bootc conversion and escalates through build/privilege.sh to get it.
@@ -79,7 +88,17 @@
79 88 # than out of the build. Everything else is matched only against what survives
80 89 # stripping these.
81 90 CONTAINERFILE_ECHO='^(STEP [0-9]+/[0-9]+:|Error: building at STEP |\+ )'
82 - NETWORK='cannot reach .* to check SHOP_REV|serves zero refs|no such host|temporary failure in name resolution|connection refused|i/o timeout|TLS handshake timeout|error pinging container registry'
91 + # `manifest unknown` and its friends are here rather than in a class of their
92 + # own, and they are the reason this line grew on 2026-08-21. Both pinned base
93 + # digests were gone from quay that day — the DEAD state refresh-base-digests.sh
94 + # is written to name — and podman died at `creating build container`, before a
95 + # single instruction ran. This script called that exit 1, "the rust-build stage
96 + # does not build at SHOP_REV af8c79c2", which is a claim about shop's revision
97 + # on evidence that never reached shop at all. A missing base is a hole in the
98 + # evidence in exactly the way an unreachable remote is, and it already has an
99 + # owner: `base-pins` reports it as a finding, and the die message below sends
100 + # you there.
101 + NETWORK='cannot reach .* to check SHOP_REV|serves zero refs|no such host|temporary failure in name resolution|connection refused|i/o timeout|TLS handshake timeout|error pinging container registry|manifest unknown|creating build container'
83 102 MISSING_LIB='could not find .* pkg-config|pkg-config .* not found|No package .* found|PKG_CONFIG_PATH|cannot find -l[a-z]'
84 103
85 104 # Exit 3, never 1: 1 is reserved for "the image does not build", and a caller
@@ -126,13 +145,23 @@
126 145 fi
127 146 }
128 147
148 + # The 2026-08-21 case: podman never got past creating the build container,
149 + # because both pinned base digests had been GC'd from quay. Copied from
150 + # /tmp/alloy-ruststage.log on astra.
151 + cat > "$tmp/dead_base_pin" <<'FIXTURE'
152 + Trying to pull quay.io/fedora/fedora@sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4...
153 + Error: creating build container: initializing source docker://quay.io/fedora/fedora@sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4: reading manifest sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4 in quay.io/fedora/fedora: manifest unknown
154 + FIXTURE
155 +
129 156 check "a real build failure is not a network failure" "$tmp/misclassified" "$NETWORK" no
157 + check "a dead base pin is an evidence hole" "$tmp/dead_base_pin" "$NETWORK" yes
158 + check "a dead base pin is not a missing lib" "$tmp/dead_base_pin" "$MISSING_LIB" no
130 159 check "an unreachable remote still reads as one" "$tmp/really_unreachable" "$NETWORK" yes
131 160 check "a real build failure is not a missing lib" "$tmp/misclassified" "$MISSING_LIB" no
132 161 check "PKG_CONFIG_PATH in a RUN line is not a probe" "$tmp/env_mentions_pkgconfig" "$MISSING_LIB" no
133 162
134 163 [ "$fails" -eq 0 ] || { printf '%d self-test failure(s)\n' "$fails" >&2; exit 1; }
135 - echo "self-test: 4 passed"
164 + echo "self-test: 6 passed"
136 165 }
137 166
138 167 case "${1:-}" in
@@ -191,6 +220,26 @@
191 220 if [ "$status" -eq 0 ]; then
192 221 echo
193 222 echo "rust-build stage builds at SHOP_REV ${SHOP_REV}"
223 +
224 + # Record what was proved, so a later bump cannot quietly leave it behind.
225 + # `crates/alloy/tests/shop_rev_verified.rs` compares the Containerfile's
226 + # `ARG SHOP_REV` against this file and fails when they differ, which turns
227 + # "run this before pushing" from an instruction into a gate. The script is
228 + # still the only thing that writes it: a stamp is evidence of a build that
229 + # happened, and hand-editing it to make a test pass is the one way to make
230 + # this worse than nothing.
231 + #
232 + # Written on every success rather than behind a flag, and the content is the
233 + # revision alone so that a run at an already-recorded rev rewrites the file
234 + # byte-for-byte. That is what keeps the nightly sweep on astra from leaving a
235 + # dirty worktree behind: the file only changes when the recorded rev is
236 + # genuinely stale, which is a state worth seeing.
237 + {
238 + echo "# The SHOP_REV that build/check-rust-stage.sh last built clean."
239 + echo "# Written by that script on success. Do not edit by hand."
240 + echo "$SHOP_REV"
241 + } > "$REPO_ROOT/build/rust-stage-verified"
242 + echo "recorded in build/rust-stage-verified"
194 243 exit 0
195 244 fi
196 245
@@ -207,6 +256,9 @@
207 256 # Reading `$out` rather than `$log` is what keeps that first pattern from
208 257 # matching the guard's own source instead of the guard firing.
209 258 if grep -qEi "$NETWORK" "$out"; then
259 + if grep -qi 'manifest unknown' "$out"; then
260 + die "the pinned base image is gone from the registry, so the build never started and this says nothing about the image. Run build/refresh-base-digests.sh --check to confirm, then without --check to move the pin"
261 + fi
210 262 die "the build could not reach something it needs (registry or git remote), so this says nothing about the image"
211 263 fi
212 264
A build/rust-stage-verified +3
@@ -1,0 +1,3 @@
1 + # The SHOP_REV that build/check-rust-stage.sh last built clean.
2 + # Written by that script on success. Do not edit by hand.
3 + af8c79c22e79f95874097bddb661ec46d8c60cb5
A crates/alloy/tests/shop_rev_verified.rs +123
@@ -1,0 +1,123 @@
1 + //! The pinned shop revision is one somebody proved the build stage can compile.
2 + //!
3 + //! `ARG SHOP_REV` is a one-line edit that looks like it cannot fail, and it
4 + //! can. shop links its system libraries rather than dlopening them, so a
5 + //! revision can arrive needing a `-devel` package the `rust-build` stage does
6 + //! not install. That is not hypothetical: shop@53551eb added `fontconfig`,
7 + //! yeslogic-fontconfig-sys probes libfontconfig through pkg-config, and
8 + //! alloy@0eb6465 moved the pin onto it. Both profiles died at `cargo install
9 + //! shop`, and the commit reached astra, srht and mnw before anyone built it.
10 + //!
11 + //! Nothing in `cargo test --workspace` touches shop: it is fetched and built
12 + //! inside the container, so the tree can be green while the image cannot be
13 + //! built at all. The check that does touch it is `build/check-rust-stage.sh`,
14 + //! which builds the stage for real. It was already written, already correct,
15 + //! and already documented as the thing to run before pushing a bump — and on
16 + //! 2026-08-09 nobody ran it.
17 + //!
18 + //! So this test does not re-derive what that script proves. It asserts only
19 + //! that the script was run at the revision the Containerfile now names: a
20 + //! clean run writes the revision into `build/rust-stage-verified`, and a bump
21 + //! that has not been checked leaves the two disagreeing.
22 + //!
23 + //! Deliberately NOT the other shape. The obvious static guard reads shop's
24 + //! manifest and asserts every dependency has its `-devel` package on the dnf
25 + //! line; `check-rust-stage.sh`'s own header rejects it, because it needs a
26 + //! crate-to-package table that is not mechanical
27 + //! (`yeslogic-fontconfig-sys` -> `fontconfig-devel`) and it passes by default
28 + //! for any dependency the table has not learned. A guard whose correctness
29 + //! rests on the same discipline that failed is not a guard. The build is the
30 + //! evidence; this file is only the record that the evidence exists.
31 + //!
32 + //! Same spirit as `build_context.rs` and `profile_split.rs`: a text check that
33 + //! runs on every `cargo test` and catches the edit a build would catch an hour
34 + //! in.
35 +
36 + use std::path::PathBuf;
37 +
38 + fn repo() -> PathBuf {
39 + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..")
40 + }
41 +
42 + fn read(relative: &str) -> String {
43 + let path = repo().join(relative);
44 + std::fs::read_to_string(&path)
45 + .unwrap_or_else(|err| panic!("cannot read {}: {err}", path.display()))
46 + }
47 +
48 + /// The revision the Containerfile pins shop to.
49 + ///
50 + /// `ARG SHOP_REV=<sha>`, first occurrence, matched at the start of a line so a
51 + /// prose mention inside a comment cannot answer instead. The script parses the
52 + /// same line the same way (`grep -oE '^ARG SHOP_REV=...'`); the two agreeing is
53 + /// the whole point of the file, so they read it alike on purpose.
54 + fn pinned_rev(containerfile: &str) -> String {
55 + containerfile
56 + .lines()
57 + .find_map(|line| line.strip_prefix("ARG SHOP_REV="))
58 + .map(|rev| rev.trim().to_string())
59 + .expect("no `ARG SHOP_REV=` line in the Containerfile")
60 + }
61 +
62 + /// The revision `build/check-rust-stage.sh` last built clean.
63 + ///
64 + /// Comment lines and blanks are skipped, so the file can say what it is to
65 + /// whoever opens it, and the first thing left is the revision.
66 + fn verified_rev(stamp: &str) -> String {
67 + stamp
68 + .lines()
69 + .map(str::trim)
70 + .find(|line| !line.is_empty() && !line.starts_with('#'))
71 + .map(str::to_string)
72 + .expect("build/rust-stage-verified names no revision")
73 + }
74 +
75 + #[test]
76 + fn shop_rev_was_built_before_it_was_pinned() {
77 + let pinned = pinned_rev(&read("Containerfile"));
78 + let verified = verified_rev(&read("build/rust-stage-verified"));
79 +
80 + assert_eq!(
81 + pinned, verified,
82 + "the Containerfile pins shop at {pinned}, and the last revision the \
83 + rust-build stage was proved to compile is {verified}.\n\n\
84 + A shop revision can need a system library the stage does not install, \
85 + and nothing else in the test suite would find out: shop is built \
86 + inside the container, not by this workspace.\n\n\
87 + Run `build/check-rust-stage.sh`. It builds the stage and records the \
88 + revision on success, which is what makes this pass. Do not edit \
89 + build/rust-stage-verified to make it pass instead — the file is a \
90 + record of a build that happened, and writing a revision nobody built \
91 + into it is worse than having no check."
92 + );
93 + }
94 +
95 + /// Both halves are shaped the way the other half expects.
96 + ///
97 + /// The assertion above compares two strings and would be satisfied by two
98 + /// equal wrong ones — an empty stamp against a Containerfile whose ARG line
99 + /// somebody reformatted, say. A full sha on each side is cheap to require and
100 + /// is what the parsers are written for.
101 + #[test]
102 + fn both_revisions_are_full_shas() {
103 + for (what, rev) in [
104 + (
105 + "the Containerfile's ARG SHOP_REV",
106 + pinned_rev(&read("Containerfile")),
107 + ),
108 + (
109 + "build/rust-stage-verified",
110 + verified_rev(&read("build/rust-stage-verified")),
111 + ),
112 + ] {
113 + assert_eq!(
114 + rev.len(),
115 + 40,
116 + "{what} is `{rev}`, which is not a 40-character sha"
117 + );
118 + assert!(
119 + rev.chars().all(|c| c.is_ascii_hexdigit()),
120 + "{what} is `{rev}`, which is not hexadecimal"
121 + );
122 + }
123 + }