Skip to main content

max / alloy

Classify on what the build said, not on what the Containerfile says podman writes the Containerfile's own source into the build log next to the build's output: once per STEP header, again inside 'Error: building at STEP "..."', and once per traced command under set -x. Every classifier here grepped that log whole, so it matched the code that would report a failure as readily as the failure itself. On 2026-08-11 the rust-stage cell hit a real failure -- two makeover versions in the shared registry, STEP 17 -- and called it a network failure, because STEP 4's header carries the SHOP_REV guard's literal 'cannot reach $url to check SHOP_REV'. The $url sits there unexpanded, which is the tell, and the expanded form is what a real failure prints. That verdict is worse than a wrong grade. The script exits 3 for network trouble, witchbroom scores the cell 'error', and an error cell is defined as knowing nothing about the code, so a genuine defect was excluded from grading rather than reported. Had the base pins not also been dead, this would have shown a broken tool while the image was unbuildable. Strip the echoed lines first and classify on the remainder. The missing -lib classifier had the same hole: PKG_CONFIG_PATH on a dnf or ENV line is source, not a probe that failed. The three patterns are named constants now, and --self-test asserts them against fixtures copied from the run that misclassified. Reintroducing the bug fails two of the four.
Author: Max Johnson <me@maxj.phd> · 2026-08-11 23:02 UTC
Signed with PGP, not checked
Commit: 6429cba606164206b538cebef9da0438d2cc9b47
Parent: 0b64bc9
1 file changed, +98 insertions, -5 deletions
M build/check-rust-stage.sh +98 -5
@@ -28,7 +28,12 @@
28 28 # left is the toolchain layer, shop, and the console.
29 29 #
30 30 # Usage:
31 - # build/check-rust-stage.sh # build it; say what happened
31 + # build/check-rust-stage.sh # build it; say what happened
32 + # build/check-rust-stage.sh --self-test # check the classifiers; builds nothing
33 + #
34 + # The self-test exists because the classifiers are the part of this script that
35 + # fails invisibly: a wrong verdict still looks like a verdict. Run it after
36 + # touching any of the three patterns.
32 37 #
33 38 # Run it after moving SHOP_REV, before pushing. The astra sweep runs it nightly
34 39 # (`rust-stage` in sweep.toml), which is the backstop for when nobody does.
@@ -59,6 +64,17 @@
59 64 # that fills a disk quietly.
60 65 IMAGE="localhost/alloy-rust-stage:check"
61 66
67 + # The three patterns this script classifies on, named so `--self-test` can
68 + # assert on them. Editing one without running the self-test is how the defect
69 + # below came back.
70 + #
71 + # CONTAINERFILE_ECHO matches lines podman wrote out of the Containerfile rather
72 + # than out of the build. Everything else is matched only against what survives
73 + # stripping these.
74 + CONTAINERFILE_ECHO='^(STEP [0-9]+/[0-9]+:|Error: building at STEP |\+ )'
75 + NETWORK='cannot reach .* to check SHOP_REV|serves zero refs|no such host|temporary failure in name resolution|connection refused|i/o timeout|TLS handshake timeout|error pinging container registry'
76 + MISSING_LIB='could not find .* pkg-config|pkg-config .* not found|No package .* found|PKG_CONFIG_PATH|cannot find -l[a-z]'
77 +
62 78 # Exit 3, never 1: 1 is reserved for "the image does not build", and a caller
63 79 # that cannot tell a broken build from a broken network learns the wrong thing
64 80 # from both.
@@ -66,8 +82,55 @@
66 82
67 83 trap 'rc=$?; [ "$rc" -eq 3 ] && exit 3; die "unexpected failure at line ${LINENO}"' ERR
68 84
85 + # Fixtures are real: every line below was copied from
86 + # ~/sweep-runs/2026-08-11T224229Z on astra, the run that misclassified.
87 + self_test() {
88 + local fails=0 tmp; tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' RETURN
89 +
90 + # The regression. STEP 4's header carries the SHOP_REV guard's own source,
91 + # with `$url` unexpanded; STEP 17 is the real failure. Classifying on the raw
92 + # log calls this a network problem and throws the finding away.
93 + cat > "$tmp/misclassified" <<'FIXTURE'
94 + STEP 4/21: RUN set -eu; url=https://makenot.work/git/max/shop.git; refs=$(git ls-remote "$url" 2>/dev/null) || { echo "cannot reach $url to check SHOP_REV" >&2; exit 1; }
95 + expected one makeover themes dir, got: /root/.cargo/registry/src/i/makeover-2.4.1/themes /root/.cargo/registry/src/i/makeover-2.5.0/themes
96 + Error: building at STEP "RUN set -eux; set -- /root/.cargo/registry/src/*/makeover-*/themes": while running runtime: exit status 1
97 + FIXTURE
98 +
99 + # The guard actually firing. The URL is expanded, which is the whole
100 + # difference between the message and the code that would print it.
101 + cat > "$tmp/really_unreachable" <<'FIXTURE'
102 + STEP 4/21: RUN set -eu; url=https://makenot.work/git/max/shop.git; || { echo "cannot reach $url to check SHOP_REV" >&2; exit 1; }
103 + cannot reach https://makenot.work/git/max/shop.git to check SHOP_REV
104 + FIXTURE
105 +
106 + # The same trap one classifier down: a dnf or ENV line naming PKG_CONFIG_PATH
107 + # is source, not a probe that failed.
108 + cat > "$tmp/env_mentions_pkgconfig" <<'FIXTURE'
109 + STEP 3/21: RUN dnf install -y pkgconf && export PKG_CONFIG_PATH=/usr/lib64/pkgconfig
110 + error[E0433]: failed to resolve: use of undeclared crate or module `foo`
111 + FIXTURE
112 +
113 + check() { # name file pattern expect(yes|no)
114 + local got=no
115 + grep -vE "$CONTAINERFILE_ECHO" "$2" | grep -qEi "$3" && got=yes
116 + if [ "$got" != "$4" ]; then
117 + printf 'self-test FAIL: %s expected %s, got %s\n' "$1" "$4" "$got" >&2
118 + fails=$((fails + 1))
119 + fi
120 + }
121 +
122 + check "a real build failure is not a network failure" "$tmp/misclassified" "$NETWORK" no
123 + check "an unreachable remote still reads as one" "$tmp/really_unreachable" "$NETWORK" yes
124 + check "a real build failure is not a missing lib" "$tmp/misclassified" "$MISSING_LIB" no
125 + check "PKG_CONFIG_PATH in a RUN line is not a probe" "$tmp/env_mentions_pkgconfig" "$MISSING_LIB" no
126 +
127 + [ "$fails" -eq 0 ] || { printf '%d self-test failure(s)\n' "$fails" >&2; exit 1; }
128 + echo "self-test: 4 passed"
129 + }
130 +
69 131 case "${1:-}" in
70 - -h|--help) sed -n '2,48p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
132 + --self-test) self_test; exit 0 ;;
133 + -h|--help) sed -n '2,55p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
71 134 "") ;;
72 135 *) die "unknown argument: $1 (see --help)" ;;
73 136 esac
@@ -82,7 +145,33 @@
82 145 printf 'building the rust-build stage; SHOP_REV=%s\n' "$SHOP_REV"
83 146
84 147 log="$(mktemp -t alloy-rust-stage.XXXXXX.log)"
85 - trap 'rm -f "$log"' EXIT
148 + # What the build said, with what the Containerfile says stripped out. See
149 + # `classify_on_output_only` below for why the two have to be separated.
150 + out="$(mktemp -t alloy-rust-stage.XXXXXX.out)"
151 + trap 'rm -f "$log" "$out"' EXIT
152 +
153 + # Podman writes the Containerfile's own source into the build log alongside the
154 + # build's output: once per `STEP n/N:` header, again inside
155 + # `Error: building at STEP "..."` when a RUN fails, and once per traced command
156 + # as `+ ...` under `set -x`. So the log holds both the failures and the code
157 + # that would report them, and a grep over the whole thing matches whichever
158 + # comes first.
159 + #
160 + # That is not hypothetical. On 2026-08-11 this cell hit a real build failure --
161 + # two makeover versions in the shared registry, STEP 17 -- and reported it as a
162 + # network failure, because STEP 4's header carries the literal
163 + # `cannot reach $url to check SHOP_REV` from the SHOP_REV guard. The `$url` sits
164 + # there unexpanded, which is the tell. The cell scored `error` instead of
165 + # `fail`, and `error` means the check learned nothing about the code, so
166 + # witchbroom excluded a genuine defect from grading entirely.
167 + #
168 + # Every classifier below therefore reads `$out`, never `$log`. A line podman
169 + # echoed out of the Containerfile is not evidence about the build, and no
170 + # wording chosen for these patterns can be safe while the source they describe
171 + # is in the same file being searched.
172 + classify_on_output_only() {
173 + grep -vE "$CONTAINERFILE_ECHO" "$log" > "$out" || true
174 + }
86 175
87 176 status=0
88 177 podman build \
@@ -100,13 +189,17 @@
100 189
101 190 # Failed. Which kind of failed is the whole value of this script: a red cell
102 191 # saying "podman exited 1" costs a person the build they are trying to avoid.
192 + classify_on_output_only
103 193
104 194 # The evidence-hole cases first. Each of these is something outside the image:
105 195 # the registry serving the pinned base, or the git remote serving shop. The
106 196 # ls-remote wording is the Containerfile's own guard talking (it prints
107 197 # `cannot reach ... to check SHOP_REV`), and the zero-refs case is the one that
108 198 # already happened once, recorded there against GoingsOn problem 49732815.
109 - if grep -qEi 'cannot reach .* to check SHOP_REV|serves zero refs|no such host|temporary failure in name resolution|connection refused|i/o timeout|TLS handshake timeout|error pinging container registry' "$log"; then
199 + #
200 + # Reading `$out` rather than `$log` is what keeps that first pattern from
201 + # matching the guard's own source instead of the guard firing.
202 + if grep -qEi "$NETWORK" "$out"; then
110 203 die "the build could not reach something it needs (registry or git remote), so this says nothing about the image"
111 204 fi
112 205
@@ -117,7 +210,7 @@
117 210 # reaching a linker or a pkg-config probe has got past the network and past
118 211 # cargo's resolver, so it is a missing header on the dnf line until proven
119 212 # otherwise.
120 - if grep -qEi 'could not find .* pkg-config|pkg-config .* not found|No package .* found|PKG_CONFIG_PATH|cannot find -l[a-z]' "$log"; then
213 + if grep -qEi "$MISSING_LIB" "$out"; then
121 214 cat <<'EOF'
122 215
123 216 It looks like a missing system library rather than a code error: a build script