Skip to main content

max / alloy

14.9 KB · 286 lines History Blame Raw
1 #!/usr/bin/env bash
2 #
3 # check-rust-stage.sh — build the Containerfile's `rust-build` stage and
4 # nothing else, to prove the image still compiles.
5 #
6 # The stage that breaks silently. It clones shop at ${SHOP_REV} and builds it
7 # with the console, so it is the one part of the image whose inputs live in
8 # another repo and move without any edit here. On 2026-08-09 shop@53551eb added
9 # `fontconfig = "0.11"`, yeslogic-fontconfig-sys links libfontconfig through
10 # pkg-config rather than dlopening it, and the stage installed
11 # `wayland-devel libxkbcommon-devel pkgconf` and nothing else. `cargo install`
12 # died on both profiles. The broken commit reached astra, srht and mnw before
13 # anyone built it, because a SHOP_REV bump is a one-line edit that looks like
14 # it cannot fail.
15 #
16 # Why a build rather than a static check. The obvious cheap guard is a test
17 # that reads shop's dependencies and asserts every one has its -devel package
18 # on the dnf line. It was rejected: it needs a crate-name to package-name table
19 # that is not mechanical (yeslogic-fontconfig-sys -> fontconfig-devel), and it
20 # passes whenever shop adds a dependency the table has not learned — a guard
21 # whose correctness depends on the same discipline that failed in the first
22 # place. Building the stage proves the property directly and needs to know
23 # nothing about what shop links. It also catches every other way this stage can
24 # break, rather than the one way it broke last.
25 #
26 # Only the builder. `--target rust-build` stops before the bootc stage, so this
27 # skips the 7 GB runtime, the browser and the profile split. What is left is the
28 # toolchain layer, shop, the house faces and the console.
29 #
30 # The faces are in scope as of 2026-08-17, and this is now the cheapest check on
31 # them: the image stopped installing fonts and started cutting them, so
32 # `quasi-type` is cloned and run in this stage. A `QUASI_TYPE_REV` that does not
33 # exist, a base whose upstream moved under its sha256, or a glyph recipe that
34 # stopped drawing all of Alloy's own codepoints each fail here rather than in a
35 # 7 GB build.
36 #
37 # Usage:
38 # build/check-rust-stage.sh # build it; say what happened
39 # build/check-rust-stage.sh --self-test # check the classifiers; builds nothing
40 #
41 # The self-test exists because the classifiers are the part of this script that
42 # fails invisibly: a wrong verdict still looks like a verdict. Run it after
43 # touching any of the three patterns.
44 #
45 # Run it after moving SHOP_REV, before pushing. That used to be an instruction
46 # and nothing else, which is how alloy@0eb6465 shipped a rev that could not
47 # build. A clean run now records the revision in `build/rust-stage-verified`,
48 # and `crates/alloy/tests/shop_rev_verified.rs` fails while the Containerfile
49 # names a different one — so the bump is refused by `cargo test` in a second,
50 # rather than by an hour of podman on someone else's machine. The astra sweep
51 # runs this script nightly (`rust-stage` in sweep.toml), which stays the
52 # backstop for a rev that stops building without anyone touching the pin.
53 #
54 # Exit codes follow refresh-base-digests.sh, for the same reason:
55 #
56 # 0 the stage builds.
57 # 1 the stage does not build. The image is unbuildable and, since the
58 # install path is "build it yourself" (docs/IMAGE.md), so is Alloy.
59 # 3 something about the run, not about the image: no podman, no network, a
60 # registry or a git remote nobody could reach, or a pinned base digest that
61 # is gone from the registry. Prints `error:`, which the sweep turns into an
62 # error cell rather than a finding. The last of those is a real defect, but
63 # it belongs to `base-pins`, which names it DEAD and exits nonzero; here it
64 # is only a build that never started.
65 #
66 # Rootless on purpose, unlike build-image.sh, which needs rootful podman for
67 # the bootc conversion and escalates through build/privilege.sh to get it.
68 # Nothing here leaves the builder stage, so the root store is not involved and
69 # a check should not ask for privilege it does not need. The
70 # consequence worth knowing: this populates the *rootless* layer cache, so it
71 # does not warm a subsequent `build/build-image.sh`.
72
73 set -Eeuo pipefail
74
75 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
76 CONTAINERFILE="$REPO_ROOT/Containerfile"
77
78 # Replaced rather than accumulated. An untagged `--target` build leaves a
79 # dangling image behind every night, and a nightly cell is exactly the thing
80 # that fills a disk quietly.
81 IMAGE="localhost/alloy-rust-stage:check"
82
83 # The three patterns this script classifies on, named so `--self-test` can
84 # assert on them. Editing one without running the self-test is how the defect
85 # below came back.
86 #
87 # CONTAINERFILE_ECHO matches lines podman wrote out of the Containerfile rather
88 # than out of the build. Everything else is matched only against what survives
89 # stripping these.
90 CONTAINERFILE_ECHO='^(STEP [0-9]+/[0-9]+:|Error: building at STEP |\+ )'
91 # `manifest unknown` and its friends are here rather than in a class of their
92 # own, and they are the reason this line grew on 2026-08-21. Both pinned base
93 # digests were gone from quay that day — the DEAD state refresh-base-digests.sh
94 # is written to name — and podman died at `creating build container`, before a
95 # single instruction ran. This script called that exit 1, "the rust-build stage
96 # does not build at SHOP_REV af8c79c2", which is a claim about shop's revision
97 # on evidence that never reached shop at all. A missing base is a hole in the
98 # evidence in exactly the way an unreachable remote is, and it already has an
99 # owner: `base-pins` reports it as a finding, and the die message below sends
100 # you there.
101 NETWORK='cannot reach .* to check SHOP_REV|serves zero refs|no such host|temporary failure in name resolution|connection refused|i/o timeout|TLS handshake timeout|error pinging container registry|manifest unknown|creating build container'
102 MISSING_LIB='could not find .* pkg-config|pkg-config .* not found|No package .* found|PKG_CONFIG_PATH|cannot find -l[a-z]'
103
104 # Exit 3, never 1: 1 is reserved for "the image does not build", and a caller
105 # that cannot tell a broken build from a broken network learns the wrong thing
106 # from both.
107 die() { printf 'error: %s\n' "$*" >&2; exit 3; }
108
109 trap 'rc=$?; [ "$rc" -eq 3 ] && exit 3; die "unexpected failure at line ${LINENO}"' ERR
110
111 # Fixtures are real: every line below was copied from
112 # ~/sweep-runs/2026-08-11T224229Z on astra, the run that misclassified.
113 self_test() {
114 local fails=0 tmp; tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' RETURN
115
116 # The regression. STEP 4's header carries the SHOP_REV guard's own source,
117 # with `$url` unexpanded; STEP 17 is the real failure. Classifying on the raw
118 # log calls this a network problem and throws the finding away.
119 cat > "$tmp/misclassified" <<'FIXTURE'
120 STEP 4/21: RUN set -eu; url=https://makenot.work/git/max/shop.git; refs=$(git ls-remote "$url" 2>/dev/null) || { echo "cannot reach $url to check SHOP_REV" >&2; exit 1; }
121 expected one makeover themes dir, got: /root/.cargo/registry/src/i/makeover-2.4.1/themes /root/.cargo/registry/src/i/makeover-2.5.0/themes
122 Error: building at STEP "RUN set -eux; set -- /root/.cargo/registry/src/*/makeover-*/themes": while running runtime: exit status 1
123 FIXTURE
124
125 # The guard actually firing. The URL is expanded, which is the whole
126 # difference between the message and the code that would print it.
127 cat > "$tmp/really_unreachable" <<'FIXTURE'
128 STEP 4/21: RUN set -eu; url=https://makenot.work/git/max/shop.git; || { echo "cannot reach $url to check SHOP_REV" >&2; exit 1; }
129 cannot reach https://makenot.work/git/max/shop.git to check SHOP_REV
130 FIXTURE
131
132 # The same trap one classifier down: a dnf or ENV line naming PKG_CONFIG_PATH
133 # is source, not a probe that failed.
134 cat > "$tmp/env_mentions_pkgconfig" <<'FIXTURE'
135 STEP 3/21: RUN dnf install -y pkgconf && export PKG_CONFIG_PATH=/usr/lib64/pkgconfig
136 error[E0433]: failed to resolve: use of undeclared crate or module `foo`
137 FIXTURE
138
139 check() { # name file pattern expect(yes|no)
140 local got=no
141 grep -vE "$CONTAINERFILE_ECHO" "$2" | grep -qEi "$3" && got=yes
142 if [ "$got" != "$4" ]; then
143 printf 'self-test FAIL: %s expected %s, got %s\n' "$1" "$4" "$got" >&2
144 fails=$((fails + 1))
145 fi
146 }
147
148 # The 2026-08-21 case: podman never got past creating the build container,
149 # because both pinned base digests had been GC'd from quay. Copied from
150 # /tmp/alloy-ruststage.log on astra.
151 cat > "$tmp/dead_base_pin" <<'FIXTURE'
152 Trying to pull quay.io/fedora/fedora@sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4...
153 Error: creating build container: initializing source docker://quay.io/fedora/fedora@sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4: reading manifest sha256:49c6900e8b5077e16de991a1732ae8a38e18ee115fa8b21dc724f7ecf50d8fe4 in quay.io/fedora/fedora: manifest unknown
154 FIXTURE
155
156 check "a real build failure is not a network failure" "$tmp/misclassified" "$NETWORK" no
157 check "a dead base pin is an evidence hole" "$tmp/dead_base_pin" "$NETWORK" yes
158 check "a dead base pin is not a missing lib" "$tmp/dead_base_pin" "$MISSING_LIB" no
159 check "an unreachable remote still reads as one" "$tmp/really_unreachable" "$NETWORK" yes
160 check "a real build failure is not a missing lib" "$tmp/misclassified" "$MISSING_LIB" no
161 check "PKG_CONFIG_PATH in a RUN line is not a probe" "$tmp/env_mentions_pkgconfig" "$MISSING_LIB" no
162
163 [ "$fails" -eq 0 ] || { printf '%d self-test failure(s)\n' "$fails" >&2; exit 1; }
164 echo "self-test: 6 passed"
165 }
166
167 case "${1:-}" in
168 --self-test) self_test; exit 0 ;;
169 -h|--help) sed -n '2,55p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
170 "") ;;
171 *) die "unknown argument: $1 (see --help)" ;;
172 esac
173
174 command -v podman >/dev/null || die "podman not found"
175 [ -f "$CONTAINERFILE" ] || die "no Containerfile at $CONTAINERFILE"
176
177 # Reported whatever happens, so a red cell names the revision it was red about
178 # without anyone opening the Containerfile at that commit.
179 SHOP_REV="$(grep -oE '^ARG SHOP_REV=[0-9a-f]{7,40}' "$CONTAINERFILE" | head -1 | cut -d= -f2 || true)"
180 [ -n "$SHOP_REV" ] || die "no ARG SHOP_REV line in $CONTAINERFILE"
181 printf 'building the rust-build stage; SHOP_REV=%s\n' "$SHOP_REV"
182
183 log="$(mktemp -t alloy-rust-stage.XXXXXX.log)"
184 # What the build said, with what the Containerfile says stripped out. See
185 # `classify_on_output_only` below for why the two have to be separated.
186 out="$(mktemp -t alloy-rust-stage.XXXXXX.out)"
187 trap 'rm -f "$log" "$out"' EXIT
188
189 # Podman writes the Containerfile's own source into the build log alongside the
190 # build's output: once per `STEP n/N:` header, again inside
191 # `Error: building at STEP "..."` when a RUN fails, and once per traced command
192 # as `+ ...` under `set -x`. So the log holds both the failures and the code
193 # that would report them, and a grep over the whole thing matches whichever
194 # comes first.
195 #
196 # That is not hypothetical. On 2026-08-11 this cell hit a real build failure --
197 # two makeover versions in the shared registry, STEP 17 -- and reported it as a
198 # network failure, because STEP 4's header carries the literal
199 # `cannot reach $url to check SHOP_REV` from the SHOP_REV guard. The `$url` sits
200 # there unexpanded, which is the tell. The cell scored `error` instead of
201 # `fail`, and `error` means the check learned nothing about the code, so
202 # witchbroom excluded a genuine defect from grading entirely.
203 #
204 # Every classifier below therefore reads `$out`, never `$log`. A line podman
205 # echoed out of the Containerfile is not evidence about the build, and no
206 # wording chosen for these patterns can be safe while the source they describe
207 # is in the same file being searched.
208 classify_on_output_only() {
209 grep -vE "$CONTAINERFILE_ECHO" "$log" > "$out" || true
210 }
211
212 status=0
213 podman build \
214 --target rust-build \
215 --pull=missing \
216 -t "$IMAGE" \
217 -f "$CONTAINERFILE" \
218 "$REPO_ROOT" 2>&1 | tee "$log" || status="${PIPESTATUS[0]}"
219
220 if [ "$status" -eq 0 ]; then
221 echo
222 echo "rust-build stage builds at SHOP_REV ${SHOP_REV}"
223
224 # Record what was proved, so a later bump cannot quietly leave it behind.
225 # `crates/alloy/tests/shop_rev_verified.rs` compares the Containerfile's
226 # `ARG SHOP_REV` against this file and fails when they differ, which turns
227 # "run this before pushing" from an instruction into a gate. The script is
228 # still the only thing that writes it: a stamp is evidence of a build that
229 # happened, and hand-editing it to make a test pass is the one way to make
230 # this worse than nothing.
231 #
232 # Written on every success rather than behind a flag, and the content is the
233 # revision alone so that a run at an already-recorded rev rewrites the file
234 # byte-for-byte. That is what keeps the nightly sweep on astra from leaving a
235 # dirty worktree behind: the file only changes when the recorded rev is
236 # genuinely stale, which is a state worth seeing.
237 {
238 echo "# The SHOP_REV that build/check-rust-stage.sh last built clean."
239 echo "# Written by that script on success. Do not edit by hand."
240 echo "$SHOP_REV"
241 } > "$REPO_ROOT/build/rust-stage-verified"
242 echo "recorded in build/rust-stage-verified"
243 exit 0
244 fi
245
246 # Failed. Which kind of failed is the whole value of this script: a red cell
247 # saying "podman exited 1" costs a person the build they are trying to avoid.
248 classify_on_output_only
249
250 # The evidence-hole cases first. Each of these is something outside the image:
251 # the registry serving the pinned base, or the git remote serving shop. The
252 # ls-remote wording is the Containerfile's own guard talking (it prints
253 # `cannot reach ... to check SHOP_REV`), and the zero-refs case is the one that
254 # already happened once, recorded there against GoingsOn problem 49732815.
255 #
256 # Reading `$out` rather than `$log` is what keeps that first pattern from
257 # matching the guard's own source instead of the guard firing.
258 if grep -qEi "$NETWORK" "$out"; then
259 if grep -qi 'manifest unknown' "$out"; then
260 die "the pinned base image is gone from the registry, so the build never started and this says nothing about the image. Run build/refresh-base-digests.sh --check to confirm, then without --check to move the pin"
261 fi
262 die "the build could not reach something it needs (registry or git remote), so this says nothing about the image"
263 fi
264
265 echo
266 echo "the rust-build stage does not build at SHOP_REV ${SHOP_REV}."
267
268 # The class this script exists for, named explicitly when it recurs. Anything
269 # reaching a linker or a pkg-config probe has got past the network and past
270 # cargo's resolver, so it is a missing header on the dnf line until proven
271 # otherwise.
272 if grep -qEi "$MISSING_LIB" "$out"; then
273 cat <<'EOF'
274
275 It looks like a missing system library rather than a code error: a build script
276 probed pkg-config, or the linker wanted a library nobody installed.
277
278 shop links its system libraries rather than dlopening them, so a SHOP_REV move
279 can add one. Its README's Building section is the list at any revision. Add the
280 Fedora -devel package to the dnf line in the rust-build stage of the
281 Containerfile, above the shop install.
282 EOF
283 fi
284
285 exit 1
286