Skip to main content

max / makenotwork

Fail the reseed when the testnot demo goes thin Phase 5 of wiki testnot-demo-content-plan, the phase that stops Phase 0 from recurring. Nine of eleven items went invisible on testnot and nothing said so for weeks. The seed's own tests passed throughout, because seed_examples.rs runs against InMemoryStorage where media always attaches and no project-level paywall hides anything, so the failure lived exactly in the gap between the test and the box. More unit tests cannot close that gap; this runs against the live site. It checks thinness rather than status codes, because every page involved returns a cheerful 200 while showing nothing. Discover is the index and each storefront is the page it points at, so the core check is the two against each other: an open storefront must render exactly what discover advertises, and a paywalled one must render nothing while still stating its count. Either side alone looks healthy while disagreeing with the other, which is precisely how the bug hid. Confirmed it would have caught the original: before the fix only the two free projects rendered anything, so the visible total was 2 against a floor of 9. Both failure paths were exercised against live testnot by tightening the floors, and both exit 1. Covers are fetched rather than counted, since a 404 cover renders as a 200 page with a broken image. The placeholder count is a sealed baseline instead of an assertion at zero: all five are still the seed's generated 16x16 PNG and real media is its own task (223f3c6f), so zero would make this red from its first run and a check that is always red is one nobody reads. Sealed, it still catches the regression that matters, a real cover reverting. Wired into the seed script's tail so the reseed and its verification cannot drift apart, with SKIP_CONTENT_SMOKE=1 for a deliberate mid-change reseed. Both scripts now have to be installed on the Sando host; the README says so.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-06 17:04 UTC
Signed with PGP, not checked
Commit: d0f7f486272a74afa00e8624e7a21fa031f37ced
Parent: 08fabb5
3 files changed, +278 insertions, -4 deletions
M sando/deploy/README.md +19 -1
@@ -14,7 +14,8 @@
14 14 | `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando.toml`). |
15 15 | `post-receive` | git remote | Push-to-deploy hook. |
16 16 | `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of every configured prod dump into `/srv/sando/backups/` (04:00 UTC). |
17 - | `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. |
17 + | `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. Ends by running the smoke check below. |
18 + | `mnw-testnot-smoke.sh` | anywhere with network | Fail loudly when the testnot demo goes thin: storefronts that render fewer items than discover advertises, covers that 404 behind a 200 page, empty feeds. Checks content, not status codes. |
18 19 | `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. |
19 20 | `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. |
20 21 | `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). |
@@ -208,13 +209,30 @@
208 209
209 210 ```sh
210 211 sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh
212 + sudo install -m 0755 mnw-testnot-smoke.sh /usr/local/bin/mnw-testnot-smoke.sh
211 213 sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/
212 214 sudo systemctl daemon-reload
213 215 ```
214 216
217 + Install both. The seed script ends by `exec`ing the smoke check from its own
218 + directory, so a seed installed without its sibling reseeds and then fails at the
219 + last step.
220 +
215 221 Reset testnot any time: `sudo /usr/local/bin/mnw-testnot-seed.sh`
216 222 (or `sudo systemctl start mnw-testnot-seed.service`).
217 223
224 + A reseed is not finished when the health check passes. Healthy is not the same
225 + as worth showing: every page can return 200 while the catalog is empty, which is
226 + how nine of eleven items stayed invisible for weeks. The smoke check is the part
227 + that notices, and it runs automatically at the end of every seed. Run it alone
228 + whenever you want the same answer: `./mnw-testnot-smoke.sh`. Set
229 + `SKIP_CONTENT_SMOKE=1` on the seed to skip it while the catalog is deliberately
230 + mid-change.
231 +
232 + Its floors are sealed baselines in the same spirit as `tests/test_hygiene.rs`:
233 + they freeze what is true today and fail on a new violation. Move one only in the
234 + improving direction, and never loosen one to make a run pass.
235 +
218 236 The seed flow is the only way testnot gets its data. The prod-restore refresh it
219 237 replaced was deleted on 2026-07-22; testnot holds no prod-derived data and there
220 238 is no path that puts any there.
M sando/deploy/mnw-testnot-seed.sh +20 -3
@@ -75,10 +75,27 @@
75 75 ts_ssh "systemctl start $SERVICE"
76 76
77 77 # Boot smoke: the app must come back healthy after the reseed.
78 + healthy=0
78 79 for _ in $(seq 1 20); do
79 80 code=$(ts_ssh "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/health" || echo 000)
80 - [ "$code" = "200" ] && { log "health OK"; exit 0; }
81 + [ "$code" = "200" ] && { log "health OK"; healthy=1; break; }
81 82 sleep 3
82 83 done
83 - echo "testnot did not return healthy after reseed" >&2
84 - exit 1
84 + if [ "$healthy" -ne 1 ]; then
85 + echo "testnot did not return healthy after reseed" >&2
86 + exit 1
87 + fi
88 +
89 + # Content smoke: healthy is not the same as worth showing. A reseed that leaves
90 + # the catalog empty returns 200 on every page, which is exactly how nine of
91 + # eleven items stayed invisible for weeks. Run the thinness check here so the
92 + # reseed and its verification cannot drift apart.
93 + #
94 + # Set SKIP_CONTENT_SMOKE=1 to reseed without it, e.g. when the catalog is
95 + # deliberately mid-change and the baselines have not been re-sealed yet.
96 + if [ "${SKIP_CONTENT_SMOKE:-0}" = "1" ]; then
97 + log "content smoke skipped (SKIP_CONTENT_SMOKE=1)"
98 + exit 0
99 + fi
100 + log "running content smoke"
101 + exec "$(dirname "$0")/mnw-testnot-smoke.sh"
A sando/deploy/mnw-testnot-smoke.sh +239
@@ -1,0 +1,239 @@
1 + #!/usr/bin/env bash
2 + # Fail loudly when the testnot.work demo goes thin.
3 + #
4 + # Phase 5 of wiki `testnot-demo-content-plan`. The failure this exists for:
5 + # nine of eleven items went invisible on testnot and nothing said so for weeks.
6 + # The seed's own tests passed the whole time, because tests/workflows/seed_examples.rs
7 + # runs against InMemoryStorage where media always attaches and no project-level
8 + # paywall hides anything. The failure lived exactly in the gap between the test
9 + # and the box, so more unit tests cannot close it. This runs against the live
10 + # site instead.
11 + #
12 + # It checks thinness, not 500s. Every page here can return a cheerful 200 while
13 + # showing nothing, and that is the state it is looking for.
14 + #
15 + # Run it after every reseed (mnw-testnot-seed.sh) and before any capture run
16 + # that photographs this box. The landing carousel will carry screenshots taken
17 + # from here, so a silent reseed that empties the catalog also silently
18 + # invalidates the marketing images.
19 + #
20 + # ./mnw-testnot-smoke.sh # check testnot.work
21 + # BASE=https://testnot.work ./mnw-testnot-smoke.sh
22 + #
23 + # Exit 0 = the demo is worth showing. Exit 1 = it is not; read the FAIL lines.
24 + set -uo pipefail
25 +
26 + BASE="${BASE:-https://testnot.work}"
27 +
28 + # ── Sealed baselines ────────────────────────────────────────────────────────
29 + #
30 + # Same idea as tests/test_hygiene.rs: freeze what is true today and fail on a
31 + # new violation, rather than demanding a clean sheet nobody will ever deliver.
32 + # A number here may only move in the improving direction, and when it does, edit
33 + # it. Never loosen one to make a run pass.
34 +
35 + # Projects the demo must carry. Below this the catalog stops reading as a
36 + # catalog and starts reading as a test fixture.
37 + MIN_PROJECTS=5
38 +
39 + # Items a visitor can actually see and click, summed across every storefront
40 + # that is not paywalled. Nine of eleven on 2026-08-06; the other two are behind
41 + # the one deliberate subscription paywall.
42 + MIN_VISIBLE_ITEMS=9
43 +
44 + # Covers still served from the seed's generated placeholder (a 16x16 grayscale
45 + # PNG, src/seed/media.rs). Five of five on 2026-08-06: the seed has never
46 + # uploaded real art, and swapping in public-domain media is its own task
47 + # (GoingsOn 223f3c6f, which also gates the landing captures 54730d4d).
48 + #
49 + # This is sealed rather than asserted at zero on purpose. Zero would make the
50 + # script red from its first run, and a check that is always red is a check
51 + # nobody reads. Sealed, it still catches the regression that matters: a project
52 + # whose real cover reverts to a placeholder pushes the count up and fails.
53 + PLACEHOLDER_COVERS_BASELINE=5
54 +
55 + # A cover at or under this is the placeholder or a broken upload, not artwork.
56 + # The placeholder is 72 bytes; real cover art is orders of magnitude larger.
57 + REAL_COVER_MIN_BYTES=2048
58 +
59 + # A page under this rendered its chrome and nothing else.
60 + MIN_PAGE_BYTES=3000
61 +
62 + TMP="$(mktemp -d)"
63 + trap 'rm -rf "$TMP"' EXIT
64 +
65 + fails=0
66 + warns=0
67 + pass() { printf ' ok %s\n' "$*"; }
68 + fail() { printf ' FAIL %s\n' "$*"; fails=$((fails + 1)); }
69 + warn() { printf ' warn %s\n' "$*"; warns=$((warns + 1)); }
70 + section() { printf '\n== %s\n' "$*"; }
71 +
72 + # Fetch $1 into $TMP/body, echo the status code.
73 + get() {
74 + curl -sS --max-time 30 -o "$TMP/body" -w '%{http_code}' "$BASE$1" 2>/dev/null || echo 000
75 + }
76 +
77 + section "reachable"
78 + code=$(get /)
79 + if [ "$code" != "200" ]; then
80 + fail "GET / returned $code; nothing else below is meaningful"
81 + printf '\n%s\n' "smoke: FAILED ($fails)"
82 + exit 1
83 + fi
84 + pass "GET / -> 200"
85 +
86 + # ── The catalog ─────────────────────────────────────────────────────────────
87 + #
88 + # Discover is the index and each storefront is the page it points at. Checking
89 + # them against each other is the whole trick: either alone can look healthy
90 + # while disagreeing with the other, which is how the paywall bug hid.
91 +
92 + section "catalog"
93 + code=$(get /discover)
94 + [ "$code" = "200" ] || fail "GET /discover returned $code"
95 +
96 + # Discover's project rows carry both the slug and the item count it advertises.
97 + grep -oE '<a href="/p/[a-z0-9-]+" class="table-row project-row">|<span class="row-items">[0-9]+</span>' "$TMP/body" |
98 + sed -e 's/.*href="\/p\///' -e 's/" class.*//' -e 's/<span class="row-items">//' -e 's/<\/span>//' |
99 + paste - - >"$TMP/advertised" 2>/dev/null
100 +
101 + n_projects=$(wc -l <"$TMP/advertised")
102 + if [ "$n_projects" -lt "$MIN_PROJECTS" ]; then
103 + fail "discover lists $n_projects projects, floor is $MIN_PROJECTS"
104 + else
105 + pass "discover lists $n_projects projects"
106 + fi
107 +
108 + visible_total=0
109 + : >"$TMP/covers"
110 +
111 + while read -r slug advertised; do
112 + [ -n "$slug" ] || continue
113 + code=$(get "/p/$slug")
114 + if [ "$code" != "200" ]; then
115 + fail "/p/$slug returned $code"
116 + continue
117 + fi
118 +
119 + rendered=$(grep -o 'class="item-card' "$TMP/body" | wc -l)
120 + grep -oE 'src="https://[^"]*/image/[^"]*"' "$TMP/body" | sed -e 's/src="//' -e 's/"$//' >>"$TMP/covers"
121 +
122 + if grep -q 'class="paywall-box"' "$TMP/body"; then
123 + # A paywalled storefront shows no items by construction: the handler
124 + # returns the paywall template before it reaches them. What it must
125 + # still do is state the count, or it contradicts the discover card that
126 + # sent the visitor here.
127 + if [ "$rendered" -ne 0 ]; then
128 + fail "/p/$slug is paywalled but rendered $rendered item cards"
129 + elif grep -qE "(^|>)[^<]*\b$advertised items? included" "$TMP/body"; then
130 + pass "/p/$slug paywalled, states its $advertised items"
131 + else
132 + fail "/p/$slug is paywalled and does not state its $advertised items; discover advertises them"
133 + fi
134 + else
135 + visible_total=$((visible_total + rendered))
136 + if [ "$rendered" -eq "$advertised" ]; then
137 + pass "/p/$slug renders $rendered items, matching discover"
138 + else
139 + fail "/p/$slug renders $rendered items, discover advertises $advertised"
140 + fi
141 + fi
142 + done <"$TMP/advertised"
143 +
144 + if [ "$visible_total" -lt "$MIN_VISIBLE_ITEMS" ]; then
145 + fail "$visible_total items visible across open storefronts, floor is $MIN_VISIBLE_ITEMS"
146 + else
147 + pass "$visible_total items visible across open storefronts"
148 + fi
149 +
150 + # ── Covers ──────────────────────────────────────────────────────────────────
151 + #
152 + # A cover URL that 404s still renders as a 200 page with a broken image, which
153 + # no status-code check sees. Fetch each one.
154 +
155 + section "covers"
156 + sort -u "$TMP/covers" >"$TMP/covers.uniq"
157 + n_covers=$(wc -l <"$TMP/covers.uniq")
158 + [ "$n_covers" -gt 0 ] || fail "no cover images referenced by any storefront"
159 +
160 + placeholders=0
161 + while read -r url; do
162 + [ -n "$url" ] || continue
163 + read -r ccode csize ctype <<<"$(curl -sS --max-time 30 -o /dev/null \
164 + -w '%{http_code} %{size_download} %{content_type}' "$url" 2>/dev/null || echo "000 0 -")"
165 + short="${url##*/projects/}"
166 + case "$ctype" in
167 + image/*) is_image=1 ;;
168 + *) is_image=0 ;;
169 + esac
170 + if [ "$ccode" != "200" ]; then
171 + fail "cover $short returned $ccode"
172 + elif [ "$is_image" -eq 0 ]; then
173 + fail "cover $short served as ${ctype:--}, not an image"
174 + elif [ "$csize" -lt "$REAL_COVER_MIN_BYTES" ]; then
175 + placeholders=$((placeholders + 1))
176 + fi
177 + done <"$TMP/covers.uniq"
178 +
179 + if [ "$placeholders" -gt "$PLACEHOLDER_COVERS_BASELINE" ]; then
180 + fail "$placeholders placeholder covers, sealed baseline is $PLACEHOLDER_COVERS_BASELINE (a real cover reverted)"
181 + elif [ "$placeholders" -eq "$PLACEHOLDER_COVERS_BASELINE" ] && [ "$placeholders" -gt 0 ]; then
182 + warn "$placeholders of $n_covers covers are still the seed placeholder (GoingsOn 223f3c6f); at baseline, not a regression"
183 + else
184 + pass "$placeholders placeholder covers, down from a baseline of $PLACEHOLDER_COVERS_BASELINE. Lower PLACEHOLDER_COVERS_BASELINE to $placeholders"
185 + fi
186 +
187 + # ── The surfaces that are not the catalog ───────────────────────────────────
188 + #
189 + # Phase 4's list. Each is a place a curious visitor lands, and each can go
190 + # blank without anything else noticing.
191 +
192 + section "other surfaces"
193 + for path in /discover?mode=items /pricing /fan-plus /docs; do
194 + code=$(get "$path")
195 + size=$(wc -c <"$TMP/body")
196 + if [ "$code" != "200" ]; then
197 + fail "$path returned $code"
198 + elif [ "$size" -lt "$MIN_PAGE_BYTES" ]; then
199 + fail "$path returned 200 but only ${size}B, which is chrome and no content"
200 + else
201 + pass "$path -> 200, ${size}B"
202 + fi
203 + done
204 +
205 + section "blogs and feeds"
206 + while read -r slug _; do
207 + [ -n "$slug" ] || continue
208 + code=$(get "/p/$slug/blog")
209 + [ "$code" = "200" ] || fail "/p/$slug/blog returned $code"
210 +
211 + # Note the path: the blog feed is feed.xml, not rss. `/p/{slug}/rss` is a
212 + # different feed, of the project's items.
213 + code=$(get "/p/$slug/blog/feed.xml")
214 + posts=$(grep -c '<item>' "$TMP/body" 2>/dev/null || echo 0)
215 + if [ "$code" != "200" ]; then
216 + fail "/p/$slug/blog/feed.xml returned $code"
217 + elif [ "$posts" -lt 1 ]; then
218 + fail "/p/$slug/blog/feed.xml is an empty feed"
219 + else
220 + pass "/p/$slug blog + feed ($posts posts)"
221 + fi
222 + done <"$TMP/advertised"
223 +
224 + # ── Deferred ────────────────────────────────────────────────────────────────
225 +
226 + section "deferred"
227 + code=$(get /library)
228 + if [ "$code" = "401" ] || [ "$code" = "302" ] || [ "$code" = "303" ]; then
229 + warn "/library is $code to anonymous, as expected. Checking it needs the seeded demo buyer from GoingsOn 839a8e5a (Phase 3, decided B, not built)"
230 + else
231 + warn "/library returned $code to an anonymous visitor; expected an auth redirect. Worth a look"
232 + fi
233 +
234 + printf '\n'
235 + if [ "$fails" -gt 0 ]; then
236 + printf 'smoke: FAILED (%d failures, %d warnings)\n' "$fails" "$warns"
237 + exit 1
238 + fi
239 + printf 'smoke: ok (%d warnings)\n' "$warns"