Skip to main content

max / makenotwork

8.5 KB · 182 lines History Blame Raw
1 #!/bin/bash
2 # Run the server's expensive suites on astra instead of on fw13.
3 #
4 # fw13 is 12 cores against 14 GB, and `~/Code/.cargo/config.toml` caps cargo at
5 # `jobs = 6` because a heavy crate costs rustc ~3 GB and the box swaps rather
6 # than compiles above that. astra is 96 cores and 125 GB, usually idle outside
7 # the 03:20 sweep. The server's integration suite is the one workload where that
8 # difference is worth a round trip: ~1,340 tests and a build of the largest crate
9 # in the tree.
10 #
11 # **This is for the expensive runs only.** A filtered local run beats this every
12 # time: `cargo test --test integration described_screens` answers in seconds,
13 # where anything here pays a sync plus a build. Reach for this when the question
14 # is "is the whole suite green", not "did my change work".
15 #
16 # WHAT IT SYNCS, AND WHY IT IS NOT JUST THE SERVER. The point of the script is
17 # testing work that is *not committed yet* -- astra's own checkout only ever has
18 # what has been pushed, which is exactly what made it useless for a day of
19 # editing quasi and the server together. So it rsyncs the working trees, and it
20 # has to send every repo the local `[patch]` block redirects, or astra silently
21 # builds the published quasi against your edited server and reports green on a
22 # combination that does not exist. That is worse than no script.
23 #
24 # CAVEAT THAT DOES NOT GO AWAY: astra is aarch64. Green here is not green on
25 # x86_64. It catches logic, not size, alignment or intrinsics assumptions.
26 #
27 # Usage:
28 # scripts/test-on-astra.sh # the integration suite
29 # scripts/test-on-astra.sh --lib # or any cargo test args
30 # scripts/test-on-astra.sh --test integration described_screens
31 #
32 # Environment:
33 # ASTRA_HOST ssh host (default: astra)
34 # ASTRA_DIR remote directory, under $HOME (default: mnw-test)
35 # ASTRA_THREADS test threads (default: 16, see the note below)
36
37 set -u
38 set -o pipefail
39
40 HOST="${ASTRA_HOST:-astra}"
41 DEST="${ASTRA_DIR:-mnw-test}"
42 # Postgres, not cores, is what bounds this suite: every test takes a
43 # `CREATE DATABASE ... TEMPLATE` clone, and astra's cluster is one instance like
44 # any other. Memory note `reference_astra_tests` records ~9/734 workflow tests
45 # hitting "connection slots reserved for SUPERUSER" under an unbounded run. So
46 # this is deliberately nowhere near 96, and raising it trades wall-clock for
47 # flakes that look like real failures.
48 THREADS="${ASTRA_THREADS:-16}"
49
50 ROOT="$(cd "$(dirname "$0")/../.." && pwd)" # ~/Code/MNW
51 CODE="$(cd "$ROOT/.." && pwd)" # ~/Code
52 CARGO_CONFIG="$CODE/.cargo/config.toml"
53
54 # Every repo that has to travel. MNW carries the server and the `shared/*` path
55 # dependencies; the rest are what `[patch]` redirects. Keep in step with
56 # `$CARGO_CONFIG` -- the check below fails the run if they drift.
57 REPOS=(MNW quasi synckit Libraries/docengine Libraries/quasi-type)
58
59 say() { printf '%s\n' "$*" >&2; }
60 die() { say "test-on-astra: $*"; exit 1; }
61
62 [ -f "$CARGO_CONFIG" ] || die "no $CARGO_CONFIG; nothing says where the siblings are"
63
64 # Drift guard. A `[patch]` entry pointing outside the synced set would build on
65 # astra from the published crate instead of the working tree, and the run would
66 # be green about code nobody has.
67 #
68 # Read through the `include`, not off the wrapper: since the per-host split the
69 # wrapper holds this box's numbers and names the shared file, so grepping it for
70 # paths finds none and the guard passes on a config it never read.
71 CONFIG_DIR="$(cd "$(dirname "$CARGO_CONFIG")" && pwd)"
72 INCLUDED="$(grep -oP '(?<=")[^"]+\.toml(?=")' "$CARGO_CONFIG" || true)"
73 [ -n "$INCLUDED" ] || die "$CARGO_CONFIG includes nothing; where is [patch]?"
74 for included in $INCLUDED; do
75 [ -f "$CONFIG_DIR/$included" ] || die "$CONFIG_DIR/$included does not exist"
76 done
77
78 missing=""
79 while read -r patched; do
80 covered=""
81 for repo in "${REPOS[@]}"; do
82 case "$patched" in "$repo"/*|"$repo") covered=1; break;; esac
83 done
84 [ -n "$covered" ] || missing="$missing $patched"
85 done < <(
86 for included in $INCLUDED; do
87 grep -oP '(?<=path = ")[^"]+' "$CONFIG_DIR/$included"
88 done | sed 's|^\.\./\.\./||' | sort -u
89 )
90 [ -z "$missing" ] || die "these patched paths are not in REPOS:$missing"
91
92 say "test-on-astra: syncing to $HOST:~/$DEST"
93 for repo in "${REPOS[@]}"; do
94 [ -d "$CODE/$repo" ] || die "$CODE/$repo does not exist"
95 ssh "$HOST" "mkdir -p ~/$DEST/$(dirname "$repo")" || die "cannot reach $HOST"
96 # --delete so a file removed locally is removed there; without it a deleted
97 # test keeps passing on astra forever. target/ and .git are excluded and
98 # therefore untouched by it, which is what keeps the remote build
99 # incremental across runs.
100 # `mutants.out/` is 218 MB of cargo-mutants detritus in synckit-client alone,
101 # which is twice everything else here put together.
102 rsync -a --delete \
103 --exclude 'target/' --exclude '.git/' --exclude 'node_modules/' \
104 --exclude 'mutants.out/' \
105 "$CODE/$repo/" "$HOST:$DEST/$repo/" \
106 || die "rsync of $repo failed"
107 done
108
109 # The patch block, resolved and made absolute.
110 #
111 # `$CARGO_CONFIG` is a symlink to `cargo-config.$(hostname -s).toml`, which
112 # carries this box's numbers and `include`s the shared file where `[patch]`
113 # actually lives (the per-host split, infra `05f5f3bc`, 2026-08-31). So sending
114 # that file is sending an `include` line and nothing else: its relative path
115 # does not exist on astra, cargo finds no patches, and the run builds the
116 # PUBLISHED quasi against the synced server -- green about a combination nobody
117 # has. That is the failure this whole block exists to prevent, and it was live
118 # from the split until 2026-09-07.
119 #
120 # Absolute paths rather than relative ones. The local pair resolves twice --
121 # `include` against the symlink's directory, then the included file's paths
122 # against the parent of its own -- and neither survives being copied to another
123 # layout. An absolute path resolves once and says where it means.
124 #
125 # `[build]` and `[profile.*]` are dropped: `jobs = 6` is a statement about
126 # fw13's 14 GB and would leave 90 of astra's cores idle.
127 REMOTE_HOME="$(ssh "$HOST" 'echo $HOME')" || die "cannot read $HOST's home"
128 for included in $INCLUDED; do
129 awk -v root="$REMOTE_HOME/$DEST" '
130 /^\[/ { keep = ($0 !~ /^\[build\]/ && $0 !~ /^\[profile/) }
131 keep {
132 gsub(/path = "\.\.\/\.\.\//, "path = \"" root "/")
133 print
134 }
135 ' "$CONFIG_DIR/$included"
136 done | ssh "$HOST" "mkdir -p ~/$DEST/.cargo && cat > ~/$DEST/.cargo/config.toml" \
137 || die "could not write the remote cargo config"
138
139 # Said out loud, because a patch that silently did not apply is the one failure
140 # a green run cannot tell you about.
141 patched="$(ssh "$HOST" "grep -c 'path = \"/' ~/$DEST/.cargo/config.toml" || echo 0)"
142 [ "$patched" -gt 0 ] || die "the remote cargo config carries no absolute patch paths"
143 say "test-on-astra: $patched patched paths, all absolute"
144
145 # `--features fast-tests` is not an optimisation to argue about: it swaps argon2
146 # from production parameters (46 MiB, 2 iterations, ~600ms) to test ones (8 MiB,
147 # 1 iteration, ~10ms), and the suite performs ~874 hashes between signup, login
148 # and admin setup. That is ~9 CPU-minutes of key derivation proving nothing. It
149 # is the only thing the feature gates; see the note on `hash_password`.
150 ARGS=("$@")
151 [ ${#ARGS[@]} -gt 0 ] || ARGS=(--test integration)
152
153 say "test-on-astra: building and running ($THREADS threads, aarch64)"
154 ssh "$HOST" bash -s -- "$DEST" "$THREADS" "${ARGS[@]}" <<'REMOTE'
155 set -u
156 DEST="$1"; shift
157 THREADS="$1"; shift
158
159 cd "$HOME/$DEST/MNW/server" || exit 1
160 # rustup's toolchain, not the distro's: /usr/bin/rustc is older than the
161 # dependency tree's floor and fails with a resolver error that names a crate
162 # rather than the compiler.
163 export PATH="$HOME/.cargo/bin:$PATH"
164 # The default 1024 is exhausted by the per-test database pool.
165 ulimit -n 65536
166 # Socket form, no host: astra's postgres has no TCP listener, and the harness's
167 # `replace_db_name` splits on the last `/`, so a `?host=` query string is
168 # mangled rather than honoured.
169 export TEST_DATABASE_URL=postgres:///postgres
170 # Verify against the committed `.sqlx` metadata rather than a live database.
171 # astra's `makenotwork` has drifted from the schema in the tree, and pointing
172 # the compile-time macros at it fails the build with errors that read like code
173 # faults. Drift in the metadata is itself a finding.
174 export SQLX_OFFLINE=true
175
176 started=$(date +%s)
177 cargo test --features fast-tests "$@" -- --test-threads="$THREADS"
178 status=$?
179 echo "test-on-astra: $(( $(date +%s) - started ))s wall on astra, aarch64" >&2
180 exit $status
181 REMOTE
182