Skip to main content

max / makenotwork

Make the two load-test scripts measure what they claim to Both were quietly broken, in ways a green run cannot report. `load-conversion-ab.sh` served one side through Askama and the other through the description layer, switching with `QUASI_SCREENS`. That switch was deleted in `64b33b26` -- `quasi::mounts` is unconditional and the route tables register no Askama counterpart -- so nothing reads the variable and every run measured the same binary four times. A delta of zero reads as "the conversion bought nothing" and means "the experiment did not run". The two sides are two builds now, at two revisions, which is also the comparison there is a question about: every described screen renders through quasi either way, and what differs is whether its markup is derived at build time and walked or built as a `Node` per request. Both trees are shipped from git rather than rsynced, so the sides differ by a revision and nothing else, and both are built and run on astra with the same database. `ssh-keys` and `analytics` are not on the seam and report in both runs, which makes them the controls: a run where they move with the seam screens measured the box. `test-on-astra.sh` shipped an empty cargo config. `~/Code/.cargo/config.toml` became a per-host wrapper on 2026-08-31 (infra `05f5f3bc`) that carries this box's numbers and `include`s the shared file where `[patch]` lives. The script read the wrapper, dropped `[build]`, and sent what was left -- an `include` line whose relative path does not exist on astra, and which its own awk dropped anyway. So there were no patches, and a run would have built the PUBLISHED quasi against the synced server, which the script's own header calls worse than no script. Its drift guard passed because it grepped the same file for paths and found none. Both now read through the `include` and rewrite the paths absolute. The local pair resolves twice -- `include` against the symlink's directory, then the included file's paths against the parent of its own -- and neither survives being copied to another layout. Both refuse to run if the config they wrote carries no absolute patch path.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session
https://claude.ai/code/session_01P8ostB2UmZJGj5WjSHRSot
Author: Max Johnson <me@maxj.phd> · 2026-09-08 03:16 UTC
Commit: 9203a969ba9c22c3197f299b620d9c81a52235ae
Parent: 68c4985
2 files changed, +178 insertions, -68 deletions
M server/scripts/load-conversion-ab.sh +128 -59
@@ -1,89 +1,158 @@
1 1 #!/usr/bin/env bash
2 2 # The alternating-pairs protocol for measuring the description layer.
3 3 #
4 - # S3 ran Askama / described / described / Askama by hand. Running it by hand is
5 - # how a measurement ends up not comparable with the one before it: the mix, the
6 - # VU count and the described set all have to be identical across the four runs,
7 - # and only the middle two differ in what serves the screens.
4 + # WHAT THE TWO SIDES ARE, AND WHY THEY CHANGED
8 5 #
9 - # Two things this answers that S3's numbers do not:
6 + # This script used to serve one side through Askama and the other through the
7 + # description layer, switching with `QUASI_SCREENS`. That switch was deleted in
8 + # `64b33b26`: `quasi::mounts` is unconditional, the route tables register no
9 + # Askama counterpart, and there is nothing left to revert to. Nothing reads the
10 + # variable, so every run the old script made measured the same binary four times
11 + # and reported a delta of zero -- which reads as "the conversion bought nothing"
12 + # and means "the experiment did not run".
10 13 #
11 - # ACCUMULATION. S3 measured ONE described screen in an otherwise-Askama mix,
12 - # and its verdict rests on database connection occupancy, which is exactly the
13 - # quantity that grows with the number of described screens. Six are converted.
14 - # Default here is all six.
14 + # So the two sides are two BUILDS, at two revisions. That is also the comparison
15 + # there is now a question about: every described screen renders through quasi
16 + # either way, and what differs is whether its markup is derived at build time
17 + # and walked (the residual seam, quasicoherent `793d99dd`) or built as a `Node`
18 + # and rendered per request.
15 19 #
16 - # THE MULTITHREADED REGIME. The two forum screens' work is an outbound HTTP
17 - # call, not a query, and the described version holds a BLOCKING-POOL thread
18 - # across it where Askama pays the same latency on a runtime worker. Sweep
19 - # MT_LATENCY to find the upstream latency at which that starts to cost
20 - # something. Sizing the pool is a config line either way, so the output worth
21 - # having is the threshold, not a pass/fail.
20 + # Both trees are built and run on astra, one after another, on the same box with
21 + # the same database. fw13 is where the day's editing happens and a load
22 + # measurement wants a quiet machine.
22 23 #
23 - # Usage:
24 - # scripts/load-conversion-ab.sh # all six screens, 0ms upstream
25 - # MT_LATENCY=500 scripts/load-conversion-ab.sh # slow Multithreaded
26 - # SCREENS=user_ssh_keys scripts/load-conversion-ab.sh # reproduce S3's shape
24 + # READ THE CONTROLS FIRST. Not every described screen is on the seam --
25 + # `ssh-keys` and `analytics` are not, for reasons filed against quasicoherent --
26 + # so they report in both runs as themselves. A run where the seam screens move
27 + # and the controls move with them measured the box, not the seam.
27 28 #
28 - # Read the report's `Rej` column before reading anything else. A rejected request
29 - # is fast, so a route answering 403 or 404 wins a latency comparison it never ran.
29 + # Also read the report's `Rej` column before anything else. A rejected request is
30 + # fast, so a route answering 403 or 404 wins a latency comparison it never ran.
30 31 #
31 32 # THE PROFILE IS RELEASE, and it is here rather than in shell history so a later
32 33 # reader knows which one produced the numbers. Render cost is the quantity being
33 - # compared, and in debug it is dominated by unoptimized rendering, so a debug
34 - # sweep measures rustc's -O0 output more than it measures the description layer.
35 - # S3's section in the wiki does not record its profile, so treat any comparison
36 - # against S3's figures as indicative rather than exact.
34 + # compared, and in debug it is dominated by unoptimized rendering.
35 + #
36 + # THE BOX IS aarch64. Absolute figures here are not comparable with the
37 + # 2026-08-14 sweep's, whatever box that used; what is comparable is the
38 + # before/after within one run of this script.
39 + #
40 + # Usage:
41 + # scripts/load-conversion-ab.sh # HEAD against the pre-seam tree
42 + # BEFORE_MNW=abc1234 BEFORE_QUASI=def5678 scripts/load-conversion-ab.sh
43 + # MT_LATENCY=500 scripts/load-conversion-ab.sh # slow Multithreaded
44 + #
45 + # Sweep MT_LATENCY to find the upstream latency at which holding a blocking-pool
46 + # thread across an outbound call starts to cost something: the two forum screens
47 + # are the only described ones whose work is an HTTP call rather than a query.
48 + # Sizing the pool is a config line either way, so the output worth having is the
49 + # threshold, not a pass/fail.
37 50
38 51 set -euo pipefail
39 52 cd "$(dirname "$0")/.."
40 53
41 - # Half the virtual users on the dashboard is far above a real day. It is chosen
42 - # so the described routes see enough concurrency to say anything at all; a
43 - # production-shaped mix puts one user on them and one user reaches no contention.
54 + CODE="$(cd ../.. && pwd)"
55 +
56 + # The tree before the seam took the screens this measures. `c554d7ed` is the
57 + # last MNW commit with four screens on the seam rather than thirteen, and
58 + # `b5a21fe` is the quasi it was written against.
59 + : "${BEFORE_MNW:=c554d7ed}"
60 + : "${BEFORE_QUASI:=b5a21fe}"
61 + : "${HOST:=astra}"
62 + : "${ROOT:=ab-seam}"
44 63 : "${MIX:=anon:25,buyer:15,creator:10,dash:50}"
45 64 : "${VUS:=60}"
46 65 : "${DURATION:=60}"
47 66 : "${RAMP:=10}"
48 - : "${SCREENS:=*}"
49 67 : "${MT_LATENCY:=0}"
50 68 : "${OUT:=target/load-ab}"
51 69
52 - export TEST_DATABASE_URL="${TEST_DATABASE_URL:-postgres:///postgres}"
53 - export LOAD_VUS="$VUS"
54 - export LOAD_DURATION_SECS="$DURATION"
55 - export LOAD_RAMP_SECS="$RAMP"
70 + AFTER_MNW="$(git -C "$CODE/MNW" rev-parse --short HEAD)"
71 + AFTER_QUASI="$(git -C "$CODE/quasi" rev-parse --short HEAD)"
72 + SHARED="$CODE/_private/infra/bootstrap/cargo-config.shared.toml"
73 +
74 + echo "before: MNW $BEFORE_MNW quasi $BEFORE_QUASI"
75 + echo "after: MNW $AFTER_MNW quasi $AFTER_QUASI"
76 +
77 + # Pristine trees from git rather than rsynced working copies. The two sides
78 + # differ by a revision and nothing else, and an uncommitted edit in either would
79 + # make the comparison say something other than what it claims.
80 + REMOTE_HOME="$(ssh "$HOST" 'echo $HOME')"
81 + ssh "$HOST" "rm -rf ~/$ROOT && mkdir -p ~/$ROOT/{before,after}/{MNW,quasi,Libraries,.cargo}"
82 +
83 + send() { git -C "$CODE/$1" archive "$2" | ssh "$HOST" "tar -x -C ~/$ROOT/$3"; }
84 + send MNW "$BEFORE_MNW" before/MNW
85 + send quasi "$BEFORE_QUASI" before/quasi
86 + send MNW "$AFTER_MNW" after/MNW
87 + send quasi "$AFTER_QUASI" after/quasi
88 +
89 + for side in before after; do
90 + # The siblings the `[patch]` block redirects that neither revision touches.
91 + for repo in synckit Libraries/docengine Libraries/quasi-type; do
92 + rsync -aq --delete --exclude 'target/' --exclude '.git/' --exclude 'mutants.out/' \
93 + "$CODE/$repo/" "$HOST:$ROOT/$side/$repo/"
94 + done
95 + # `[patch]` with absolute paths. The local pair resolves twice -- `include`
96 + # against the symlink's directory, then the included file's paths against
97 + # the parent of its own -- and neither survives being copied to another
98 + # layout. `[build]` and `[profile]` are fw13's numbers.
99 + awk -v root="$REMOTE_HOME/$ROOT/$side" '
100 + /^\[/ { keep = ($0 ~ /^\[patch/) }
101 + keep { gsub(/path = "\.\.\/\.\.\//, "path = \"" root "/"); print }
102 + ' "$SHARED" | ssh "$HOST" "cat > ~/$ROOT/$side/.cargo/config.toml"
103 + done
104 +
105 + # A patch that silently did not apply is the one failure a green run cannot tell
106 + # you about: the side would be built against the published quasi rather than the
107 + # revision this script just shipped.
108 + for side in before after; do
109 + [ "$(ssh "$HOST" "grep -c 'path = \"/' ~/$ROOT/$side/.cargo/config.toml")" -gt 0 ] ||
110 + { echo "the $side cargo config carries no absolute patch paths" >&2; exit 1; }
111 + done
112 +
113 + ssh "$HOST" "cat > ~/$ROOT/run.sh" <<REMOTE
114 + set -u
115 + ROOT="\$HOME/$ROOT"
116 + OUT="\$ROOT/out"
117 + mkdir -p "\$OUT"
118 + export PATH="\$HOME/.cargo/bin:\$PATH"
119 + export TEST_DATABASE_URL="\${TEST_DATABASE_URL:-postgres:///postgres}"
120 + export LOAD_VUS=$VUS
121 + export LOAD_DURATION_SECS=$DURATION
122 + export LOAD_RAMP_SECS=$RAMP
56 123 export LOAD_MIX="$MIX"
57 - export LOAD_MT_LATENCY_MS="$MT_LATENCY"
124 + export LOAD_MT_LATENCY_MS=$MT_LATENCY
58 125
59 - mkdir -p "$OUT"
60 -
61 - # Built once, outside the timed runs: a cold compile inside run 1 would show up
62 - # as run 1 being slower than run 4, which is the shape the alternation exists to
63 - # cancel out.
64 - echo "Building the load binary (release)..."
65 - cargo test --release --test load --no-run --quiet
126 + # Built outside the timed runs: a cold compile inside run 1 would show up as run
127 + # 1 being slower than run 4, which is the shape the alternation cancels out.
128 + for side in before after; do
129 + echo "=== building \$side ==="
130 + ( cd "\$ROOT/\$side/MNW/server" && cargo test --release --test load --no-run --quiet ) \
131 + > "\$OUT/build-\$side.log" 2>&1 || { echo "build \$side FAILED"; exit 1; }
132 + done
66 133
67 134 run() {
68 - local label="$1"
69 - local screens="$2"
70 - local path="$OUT/$label.txt"
71 - echo
72 - echo "=== $label (QUASI_SCREENS=${screens:-<none>}) ==="
73 - QUASI_SCREENS="$screens" \
74 - cargo test --release --test load -- --ignored --nocapture 2>&1 | tee "$path" |
75 - sed -n '/LOAD TEST REPORT/,$p'
135 + echo "=== run \$1 (\$2) ==="
136 + ( cd "\$ROOT/\$2/MNW/server" && cargo test --release --test load -- --ignored --nocapture ) \
137 + > "\$OUT/\$1-\$2.txt" 2>&1
138 + echo "run \$1 exit=\$?"
76 139 }
77 140
78 - # Askama, described, described, Askama. The pairs are inner and the controls are
79 - # outer so a drift in the box over the run (thermal, page cache, another process
80 - # arriving) lands on both sides rather than on one.
81 - run 1-askama ""
82 - run 2-described "$SCREENS"
83 - run 3-described "$SCREENS"
84 - run 4-askama ""
141 + # before, after, after, before. The pairs are inner and the controls outer so a
142 + # drift in the box over the run lands on both sides rather than on one.
143 + run 1 before
144 + run 2 after
145 + run 3 after
146 + run 4 before
147 + echo DONE
148 + REMOTE
85 149
150 + echo "Running on $HOST. Four reports will land in ~/$ROOT/out/."
151 + ssh "$HOST" "bash ~/$ROOT/run.sh"
152 +
153 + mkdir -p "$OUT"
154 + scp -q "$HOST:$ROOT/out/*.txt" "$OUT/" || true
86 155 echo
87 - echo "Four reports under $OUT/. Compare 2+3 against 1+4, per endpoint label."
88 - echo "The described screens report under 'HTMX described:*' in every run: same"
89 - echo "address either way, so the rows line up and only what serves them changed."
156 + echo "Reports under $OUT/. Compare 2+3 against 1+4, per endpoint label."
157 + echo "The described screens report under 'HTMX described:*' in both, so the rows"
158 + echo "line up and only what serves them changed."
M server/scripts/test-on-astra.sh +50 -9
@@ -64,6 +64,17 @@
64 64 # Drift guard. A `[patch]` entry pointing outside the synced set would build on
65 65 # astra from the published crate instead of the working tree, and the run would
66 66 # be green about code nobody has.
67 + #
68 + # Read through the `include`, not off the wrapper: since the per-host split the
69 + # wrapper holds this box's numbers and names the shared file, so grepping it for
70 + # paths finds none and the guard passes on a config it never read.
71 + CONFIG_DIR="$(cd "$(dirname "$CARGO_CONFIG")" && pwd)"
72 + INCLUDED="$(grep -oP '(?<=")[^"]+\.toml(?=")' "$CARGO_CONFIG" || true)"
73 + [ -n "$INCLUDED" ] || die "$CARGO_CONFIG includes nothing; where is [patch]?"
74 + for included in $INCLUDED; do
75 + [ -f "$CONFIG_DIR/$included" ] || die "$CONFIG_DIR/$included does not exist"
76 + done
77 +
67 78 missing=""
68 79 while read -r patched; do
69 80 covered=""
@@ -71,7 +82,11 @@
71 82 case "$patched" in "$repo"/*|"$repo") covered=1; break;; esac
72 83 done
73 84 [ -n "$covered" ] || missing="$missing $patched"
74 - done < <(grep -oP '(?<=path = ")[^"]+' "$CARGO_CONFIG" | sort -u)
85 + done < <(
86 + for included in $INCLUDED; do
87 + grep -oP '(?<=path = ")[^"]+' "$CONFIG_DIR/$included"
88 + done | sed 's|^\.\./\.\./||' | sort -u
89 + )
75 90 [ -z "$missing" ] || die "these patched paths are not in REPOS:$missing"
76 91
77 92 say "test-on-astra: syncing to $HOST:~/$DEST"
@@ -91,16 +106,42 @@
91 106 || die "rsync of $repo failed"
92 107 done
93 108
94 - # The patch block, verbatim, because its paths are relative to the config file's
95 - # own parent and the layout above mirrors ~/Code exactly. `[build]` and
96 - # `[profile.*]` are dropped: `jobs = 6` is a statement about fw13's 14 GB and
97 - # would leave 90 of astra's cores idle.
98 - awk '
99 - /^\[/ { keep = ($0 !~ /^\[build\]/ && $0 !~ /^\[profile/) }
100 - keep { print }
101 - ' "$CARGO_CONFIG" | ssh "$HOST" "mkdir -p ~/$DEST/.cargo && cat > ~/$DEST/.cargo/config.toml" \
109 + # The patch block, resolved and made absolute.
110 + #
111 + # `$CARGO_CONFIG` is a symlink to `cargo-config.$(hostname -s).toml`, which
112 + # carries this box's numbers and `include`s the shared file where `[patch]`
113 + # actually lives (the per-host split, infra `05f5f3bc`, 2026-08-31). So sending
114 + # that file is sending an `include` line and nothing else: its relative path
115 + # does not exist on astra, cargo finds no patches, and the run builds the
116 + # PUBLISHED quasi against the synced server -- green about a combination nobody
117 + # has. That is the failure this whole block exists to prevent, and it was live
118 + # from the split until 2026-09-07.
119 + #
120 + # Absolute paths rather than relative ones. The local pair resolves twice --
121 + # `include` against the symlink's directory, then the included file's paths
122 + # against the parent of its own -- and neither survives being copied to another
123 + # layout. An absolute path resolves once and says where it means.
124 + #
125 + # `[build]` and `[profile.*]` are dropped: `jobs = 6` is a statement about
126 + # fw13's 14 GB and would leave 90 of astra's cores idle.
127 + REMOTE_HOME="$(ssh "$HOST" 'echo $HOME')" || die "cannot read $HOST's home"
128 + for included in $INCLUDED; do
129 + awk -v root="$REMOTE_HOME/$DEST" '
130 + /^\[/ { keep = ($0 !~ /^\[build\]/ && $0 !~ /^\[profile/) }
131 + keep {
132 + gsub(/path = "\.\.\/\.\.\//, "path = \"" root "/")
133 + print
134 + }
135 + ' "$CONFIG_DIR/$included"
136 + done | ssh "$HOST" "mkdir -p ~/$DEST/.cargo && cat > ~/$DEST/.cargo/config.toml" \
102 137 || die "could not write the remote cargo config"
103 138
139 + # Said out loud, because a patch that silently did not apply is the one failure
140 + # a green run cannot tell you about.
141 + patched="$(ssh "$HOST" "grep -c 'path = \"/' ~/$DEST/.cargo/config.toml" || echo 0)"
142 + [ "$patched" -gt 0 ] || die "the remote cargo config carries no absolute patch paths"
143 + say "test-on-astra: $patched patched paths, all absolute"
144 +
104 145 # `--features fast-tests` is not an optimisation to argue about: it swaps argon2
105 146 # from production parameters (46 MiB, 2 iterations, ~600ms) to test ones (8 MiB,
106 147 # 1 iteration, ~10ms), and the suite performs ~874 hashes between signup, login