max / makenotwork

pom: version astra's test-runner setup The scheduler landed with nothing on disk to describe what it runs. The runner script and the systemd drop-in existed only on astra, untracked by anything, which is the same shape of problem as the staging tree that nothing recorded. run-ci.sh carries the four environment facts that each produce a failure looking like something else: staging mirrors the ~/Code tree because the Tauri apps symlink across it, PGUSER is required because sqlx falls back to whoami's "anonymous" inside the sandbox, the harnesses need TEST_DATABASE_URL because they default to TCP against a socket-only postgres, and cargo's job count needs capping because 96 concurrent rustc processes exhaust the cgroup before the build finishes. pom-astra.toml had drifted: no [targets.mnw.tests] block at all, and every other block still naming the SSH hop that never worked. The peer token stays out, as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-06 21:41 UTC

Signed with PGP, not checked

Commit: 9520ad2ef9f69961e63f20e1ad176163970e8f34

Parent: 9cc042b

4 files changed, +209 insertions, -15 deletions

M pom/deploy/README.md +46

			@@ -64,6 +64,52 @@
64	64		unit), so it is a script-guarded grant rather than a broad
65	65		`install` + `systemctl` one.
66	66
	67	+	## The test runner on astra
	68	+
	69	+	astra is the only instance that runs test suites, and it runs them itself rather
	70	+	than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can
	71	+	see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop
	72	+	to reach (Tailscale SSH does not intercept a node connecting to itself, which is
	73	+	why the config's old `ssh = "max@<astra>"` failed `Connection refused` and no
	74	+	suite ever ran). `TestsConfig.ssh` is therefore optional; omitted, the command
	75	+	runs as a local child.
	76	+
	77	+	Everything the runner needs lives under `/var/lib/pom`, the one path the hardened
	78	+	unit can write:
	79	+
	80	+	\| What \| Where \| Source \|
	81	+	\|------\|-------\|--------\|
	82	+	\| Runner script \| `/var/lib/pom/staging/run-ci.sh` \| `run-ci.sh` here \|
	83	+	\| Unit overrides \| `/etc/systemd/system/pom.service.d/10-test-runner.conf` \| `pom.service.d-10-test-runner.conf` here \|
	84	+	\| Clones \| `/var/lib/pom/staging/{MNW,synckit,Apps/*}` \| cloned from `/home/max/git-mirrors` \|
	85	+	\| Toolchain \| `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` \| rustup, as the `pom` user, plus `sqlx-cli` \|
	86	+
	87	+	The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any
	88	+	cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the
	89	+	clones can fetch from the mirrors.
	90	+
	91	+	`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records
	92	+	that layout, and both Tauri apps carry a symlink that reaches across it
	93	+	(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`).
	94	+	Flatten the clones and it dangles, and the app's `build.rs` panics on a read.
	95	+
	96	+	Postgres notes, all of which cause failures that look like something else:
	97	+	`PGUSER=pom` is required because sqlx cannot resolve a username inside the
	98	+	sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need
	99	+	`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is
	100	+	socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than
	101	+	compiling against the committed `.sqlx` cache, which goes stale silently whenever
	102	+	a migration lands without a `cargo sqlx prepare`.
	103	+
	104	+	To reproduce a red suite by hand:
	105	+
	106	+	```
	107	+	sudo -u pom /var/lib/pom/staging/run-ci.sh mnw
	108	+	```
	109	+
	110	+	A failure that only appears under `systemd-run` with the unit's properties is a
	111	+	hardening problem, not a test problem.
	112	+
67	113		## Rollback
68	114
69	115		The installer keeps the previous binary as `<install-path>.prev`. Rolling back

M pom/deploy/pom-astra.toml +15 -15

			@@ -48,6 +48,11 @@
48	48		[targets.mnw.tls]
49	49		host = "makenot.work"
50	50
	51	+	[targets.mnw.tests]
	52	+	command = "/var/lib/pom/staging/run-ci.sh mnw"
	53	+	timeout_secs = 5400
	54	+	staleness_days = 7
	55	+
51	56		[targets.mt]
52	57		label = "Multithreaded Forum"
53	58		expected_routes = ["/"]
			@@ -64,9 +69,8 @@
64	69		host = "forums.makenot.work"
65	70
66	71		[targets.mt.tests]
67		-	ssh = "max@100.106.221.39"
68		-	command = "cd /home/max/staging/multithreaded && cargo test --workspace 2>&1"
69		-	timeout_secs = 300
	72	+	command = "/var/lib/pom/staging/run-ci.sh mt"
	73	+	timeout_secs = 2700
70	74		staleness_days = 7
71	75
72	76		[targets.mnw-cli]
			@@ -112,36 +116,32 @@
112	116		label = "GoingsOn"
113	117
114	118		[targets.go.tests]
115		-	ssh = "max@100.106.221.39"
116		-	command = "cd /home/max/staging/goingson && cargo test --workspace 2>&1"
117		-	timeout_secs = 600
	119	+	command = "/var/lib/pom/staging/run-ci.sh go"
	120	+	timeout_secs = 2700
118	121		staleness_days = 7
119	122
120	123		[targets.bb]
121	124		label = "Balanced Breakfast"
122	125
123	126		[targets.bb.tests]
124		-	ssh = "max@100.106.221.39"
125		-	command = "cd /home/max/staging/balanced_breakfast && cargo test --workspace 2>&1"
126		-	timeout_secs = 600
	127	+	command = "/var/lib/pom/staging/run-ci.sh bb"
	128	+	timeout_secs = 2700
127	129		staleness_days = 7
128	130
129	131		[targets.af]
130	132		label = "AudioFiles"
131	133
132	134		[targets.af.tests]
133		-	ssh = "max@100.106.221.39"
134		-	command = "cd /home/max/staging/audiofiles && cargo test --workspace 2>&1"
135		-	timeout_secs = 600
	135	+	command = "/var/lib/pom/staging/run-ci.sh af"
	136	+	timeout_secs = 2700
136	137		staleness_days = 7
137	138
138	139		[targets.sk]
139	140		label = "SyncKit SDK"
140	141
141	142		[targets.sk.tests]
142		-	ssh = "max@100.106.221.39"
143		-	command = "cd /home/max/staging/synckit-client && cargo test 2>&1"
144		-	timeout_secs = 300
	143	+	command = "/var/lib/pom/staging/run-ci.sh sk"
	144	+	timeout_secs = 1200
145	145		staleness_days = 7
146	146
147	147		[peers.hetzner]

A pom/deploy/pom.service.d-10-test-runner.conf +22

		@@ -1,0 +1,22 @@
1	+	# Room for the test suites PoM now schedules.
2	+	#
3	+	# The base unit caps the whole cgroup at 256M, which is generous for the daemon
4	+	# and nowhere near a cargo build. The suites run as child processes in this same
5	+	# cgroup, so the cap applies to them: left at 256M every scheduled run would be
6	+	# OOM-killed and land in the dashboard as a failing suite. astra has 125G.
7	+	#
8	+	# 48G rather than 16G because the MNW server workspace blew through 16G and 20G
9	+	# of swap before dying. run-ci.sh also caps cargo's job count: 96 cores means 96
10	+	# concurrent rustc processes by default, and the memory ceiling is the wrong
11	+	# place to discover that.
12	+	#
13	+	# ProtectHome drops to read-only rather than staying true because the staging
14	+	# clones fetch from the mirrors under /home/max/git-mirrors. Read-only is the
15	+	# whole requirement: nothing here writes outside /var/lib/pom.
16	+	[Service]
17	+	MemoryMax=48G
18	+	ProtectHome=read-only
19	+	Environment=HOME=/var/lib/pom
20	+	Environment=CARGO_HOME=/var/lib/pom/.cargo
21	+	Environment=RUSTUP_HOME=/var/lib/pom/.rustup
22	+	LimitNOFILE=65536

A pom/deploy/run-ci.sh +126

		@@ -1,0 +1,126 @@
1	+	#!/bin/sh
2	+	# Test runner for PoM's scheduled suites on astra.
3	+	#
4	+	# pom.service calls this once a target's last run goes stale. Everything it
5	+	# touches lives under /var/lib/pom, the one path the hardened unit can write:
6	+	# the clones, the rust toolchain, and the build artifacts. That is deliberate.
7	+	# The daemon runs as `pom` with ProtectHome, so max's checkouts and max's
8	+	# toolchain are out of reach by design, and this script must not depend on them.
9	+	#
10	+	# The one exception is the git mirrors it fetches from, which are read-only.
11	+	#
12	+	# Usage: run-ci.sh <target> [filter]
13	+	# target one of: mnw mt go bb af sk
14	+	# filter optional cargo test filter, passed through by PoM
15	+	#
16	+	# Exit status is the suite's, which is what PoM records as pass/fail.
17	+
18	+	set -eu
19	+
20	+	STAGING="/var/lib/pom/staging"
21	+	MIRRORS="/home/max/git-mirrors"
22	+
23	+	# The pom user's own toolchain. /usr/bin/cargo is the system rustc, which
24	+	# trails what these dependency trees need.
25	+	export HOME="/var/lib/pom"
26	+	export RUSTUP_HOME="/var/lib/pom/.rustup"
27	+	export CARGO_HOME="/var/lib/pom/.cargo"
28	+	export PATH="$CARGO_HOME/bin:$PATH"
29	+
30	+	# Parallel test databases exhaust the default 1024 descriptors.
31	+	ulimit -n 65536 2>/dev/null \|\| true
32	+
33	+	# Peer auth on the socket means postgres wants an OS username, and sqlx cannot
34	+	# work one out inside the service sandbox: it falls back to the `whoami` crate's
35	+	# placeholder and every connection is refused as user "anonymous". libpq tools
36	+	# and sqlx both honour PGUSER, so say it once here.
37	+	export PGUSER=pom
38	+
39	+	# astra has 96 cores, and cargo's default job count is all of them. Ninety-six
40	+	# concurrent rustc processes on the MNW server workspace peaked past 16G and 20G
41	+	# of swap before the cgroup killed it. This is a memory cap wearing a job-count
42	+	# costume; the wall-clock cost is small next to thrashing.
43	+	export CARGO_BUILD_JOBS="${CARGO_BUILD_JOBS:-16}"
44	+
45	+	# The mirrors are max's, this runs as pom, and git refuses a repo it considers
46	+	# foreign. Without this every fetch below fails and the clones silently test
47	+	# whatever commit they were created at, forever. Scoped to this process rather
48	+	# than written into a global gitconfig.
49	+	GIT_CONFIG_COUNT=1
50	+	GIT_CONFIG_KEY_0=safe.directory
51	+	GIT_CONFIG_VALUE_0='*'
52	+	export GIT_CONFIG_COUNT GIT_CONFIG_KEY_0 GIT_CONFIG_VALUE_0
53	+
54	+	target="${1:-}"
55	+	filter="${2:-}"
56	+
57	+	# The layout under $STAGING mirrors ~/Code on purpose, Apps/ included. Nothing
58	+	# in git records that tree, so a plain clone does not reproduce it, and the
59	+	# Tauri apps have a symlink that reaches across it:
60	+	# Apps/<app>/src-tauri/frontend/js/shared-updater.js
61	+	# -> ../../../../../MNW/shared/tauri-updater-ui/updater.js
62	+	# Flatten the tree and that dangles, and the app's build.rs panics on a read.
63	+	case "$target" in
64	+	mnw) dir="$STAGING/MNW/server" ;;
65	+	mt) dir="$STAGING/MNW/multithreaded" ;;
66	+	go) dir="$STAGING/Apps/goingson" ;;
67	+	bb) dir="$STAGING/Apps/balanced_breakfast" ;;
68	+	af) dir="$STAGING/Apps/audiofiles" ;;
69	+	sk) dir="$STAGING/synckit/synckit-client" ;;
70	+	*)
71	+	echo "run-ci.sh: unknown target '${target}' (want: mnw mt go bb af sk)" >&2
72	+	exit 2
73	+	;;
74	+	esac
75	+
76	+	if [ ! -d "$dir" ]; then
77	+	echo "run-ci.sh: $dir does not exist, clone it from $MIRRORS first" >&2
78	+	exit 2
79	+	fi
80	+
81	+	# Track the mirror so a run tests current main rather than whatever was cloned
82	+	# months ago. A failed fetch is not fatal: testing a slightly old tree beats
83	+	# reporting nothing, and the staleness clock keeps ticking either way.
84	+	repo_root=$(cd "$dir" && git rev-parse --show-toplevel)
85	+	if ! (cd "$repo_root" && git fetch --quiet origin && git reset --quiet --hard origin/HEAD); then
86	+	echo "run-ci.sh: warning, could not update $repo_root, testing the existing checkout" >&2
87	+	fi
88	+
89	+	cd "$dir"
90	+	echo "run-ci.sh: $target at $(git rev-parse --short HEAD) in $dir"
91	+
92	+	# Postgres on astra listens on a unix socket only, so the URLs carry no host.
93	+	# Both harnesses default to a TCP URL, which fails to connect here and fails
94	+	# every test in the suite within seconds rather than failing loudly once.
95	+	# Do not add ?host=... : the harnesses split these on the last '/'.
96	+	# Auth is peer on the socket, so the `pom` role needs LOGIN and CREATEDB.
97	+	case "$target" in
98	+	mnw)
99	+	# Its own database, not max's `makenotwork`: the sqlx macros are
100	+	# checked at compile time against whatever DATABASE_URL points at, so
101	+	# CI needs a schema it owns and can migrate. Compiling against the
102	+	# committed .sqlx cache instead is not an option worth taking, since
103	+	# the cache is hand-maintained by `cargo sqlx prepare` and goes stale
104	+	# silently the moment a migration lands without one.
105	+	DATABASE_URL='postgres:///pom_ci_makenotwork'
106	+	TEST_DATABASE_URL='postgres:///postgres'
107	+	export DATABASE_URL TEST_DATABASE_URL
108	+
109	+	psql "$TEST_DATABASE_URL" -tAc \
110	+	"SELECT 1 FROM pg_database WHERE datname = 'pom_ci_makenotwork'" \
111	+	\| grep -q 1 \|\| createdb pom_ci_makenotwork
112	+	sqlx migrate run --source migrations --database-url "$DATABASE_URL"
113	+	;;
114	+	mt)
115	+	TEST_DATABASE_URL='postgres:///postgres'
116	+	export TEST_DATABASE_URL
117	+	;;
118	+	esac
119	+
120	+	# --test-threads=4 rather than the default: at full parallelism the suites
121	+	# intermittently exhaust postgres connection slots on this box.
122	+	if [ -n "$filter" ]; then
123	+	exec cargo test --workspace "$filter" -- --test-threads=4
124	+	else
125	+	exec cargo test --workspace -- --test-threads=4
126	+	fi