Skip to main content

max / makenotwork

seed: Phase 6 — testnot seed reset flow replaces prod-refresh Add mnw-testnot-seed.sh: reset testnot's schema and reseed it from --seed-examples instead of restoring a production backup, so the staging box holds no prod-derived data. It stops the app, drops+recreates the schema (the refresh script's proven public-owner block, verbatim), runs the binary once with --seed-examples (migrates the empty schema, runs the guarded seed, exits before binding a port), restarts the app, and smokes /health. Idempotent, no pause-flag. Add a oneshot mnw-testnot-seed.service (no timer — the catalog is fixed, reseed on demand). Rewrite the deploy README's testnot section: seed as source of truth, install + cutover sequence, rollback to the prod-mirror refresh (whose units are kept), and the S3_*/MinIO prerequisite for media. Scripts only; the live-box cutover is operational (gated on MinIO) and documented in the runbook, not executed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-11 23:13 UTC
Commit: 7a3d9d2f36679505fe1c822e4cb7a3b6625fd742
Parent: c4ca578
3 files changed, +136 insertions, -18 deletions
M sando/deploy/README.md +36 -18
@@ -14,7 +14,8 @@ live in the Syncthing private layer (`_private/infra/`, `_private/deploy`).
14 14 | `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando.toml`). |
15 15 | `post-receive` | git remote | Push-to-deploy hook. |
16 16 | `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of the prod backup to `/srv/sando/backups/latest.sql.gz` (04:00 UTC). |
17 - | `mnw-testnot-refresh.{sh,service,timer}` | Sando host | Daily refresh of the testnot.work staging mirror (05:00 UTC). |
17 + | `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. |
18 + | `mnw-testnot-refresh.{sh,service,timer}` | Sando host | Prod-restore refresh of testnot (05:00 UTC). **Superseded by the seed flow; kept for rollback.** |
18 19 | `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. |
19 20 | `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. |
20 21 | `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). |
@@ -115,32 +116,49 @@ After that, every promote that reaches the node ships the server and its
115 116 companions together; no separate `mnw-cli` deploy step. The legacy
116 117 `mnw-cli/deploy/deploy.sh` is retired once the first lockstep prod promote lands.
117 118
118 - ## testnot.work staging mirror
119 + ## testnot.work staging
119 120
120 - testnot is a read-only mirror of production, gated app-side to Fan+/creator
121 - accounts (`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+
122 - members can preview upcoming features, and to back the pre-cutover migration
123 - dry-run.
121 + testnot is gated app-side to Fan+/creator accounts
122 + (`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+ members can
123 + preview upcoming features, and to back the pre-cutover migration dry-run.
124 124
125 - **Daily refresh** (`mnw-testnot-refresh.timer` → `.service` → `.sh`): reloads
126 - testnot's database from the backup `sandod-backup-fetch` already pulls, so the
127 - mirror tracks live. The script stops the app, resets the schema (recreating
128 - `public` owned by the app role — PG15+ otherwise blocks the app role's boot
129 - migrations), restores the dump as the postgres superuser over Tailscale SSH,
130 - and restarts the app, which applies any newer migrations on boot. Writes
131 - between refreshes are non-durable by design.
125 + **Source of truth: the seeded example catalog** (`mnw-testnot-seed.sh`). testnot
126 + no longer mirrors production. Instead it runs a self-contained catalog of
127 + fabricated `@example.test` creators and public-domain items, built by the app's
128 + `--seed-examples` flow — so no prod-derived data lives on the staging box. The
129 + script stops the app, resets the schema (recreating `public` owned by the app
130 + role — PG15+ otherwise blocks the app role's boot migrations), then runs the
131 + binary once with `--seed-examples`, which migrates the empty schema and runs the
132 + guarded seed before exiting, and restarts the app. It is idempotent — re-run it
133 + any time to reset testnot to the fixed catalog. There is no pause-flag; the
134 + catalog is stable by construction.
135 +
136 + Media (previews/downloads) attaches only when object storage is configured in
137 + `/etc/mnw/makenotwork.env` (`S3_*`, plus `S3_PUBLIC_BUCKET` + `CDN_BASE_URL` for
138 + covers). Until MinIO is stood up on testnot, items seed **hidden** (creators,
139 + projects, blog posts, and follow counts still show).
132 140
133 141 Install on the Sando host:
134 142
135 143 ```sh
136 - sudo install -m 0755 mnw-testnot-refresh.sh /usr/local/bin/mnw-testnot-refresh.sh
137 - sudo install -m 0644 mnw-testnot-refresh.service /etc/systemd/system/
138 - sudo install -m 0644 mnw-testnot-refresh.timer /etc/systemd/system/
144 + sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh
145 + sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/
139 146 sudo systemctl daemon-reload
140 - sudo systemctl enable --now mnw-testnot-refresh.timer
141 147 ```
142 148
143 - Run a refresh manually: `sudo /usr/local/bin/mnw-testnot-refresh.sh`
149 + Cutover (prod-mirror → example catalog):
150 +
151 + ```sh
152 + sudo /usr/local/bin/mnw-testnot-seed.sh # reseed the box
153 + sudo systemctl disable --now mnw-testnot-refresh.timer # stop the daily prod restore
154 + ```
155 +
156 + Reset testnot any time afterward: `sudo /usr/local/bin/mnw-testnot-seed.sh`
157 + (or `sudo systemctl start mnw-testnot-seed.service`).
158 +
159 + **Rollback** to the prod-mirror flow: `sudo systemctl enable --now
160 + mnw-testnot-refresh.timer` (its units are kept installed), then run
161 + `sudo /usr/local/bin/mnw-testnot-refresh.sh` once to reload from the backup.
144 162
145 163 **Redeploy the binary** (no Sando integration yet — manual): build the release
146 164 on the Sando host, then over Tailscale SSH as root on the target, copy the
A sando/deploy/mnw-testnot-seed.service +20
@@ -0,0 +1,20 @@
1 + # One-shot: reset testnot.work to the fabricated example catalog.
2 + #
3 + # Replaces the daily mnw-testnot-refresh.timer/.service (prod-restore). The seed
4 + # catalog is fixed, so this is NOT scheduled — run it deliberately on cutover or
5 + # to reset the box: systemctl start mnw-testnot-seed.service (or run the script
6 + # directly). The mnw-testnot-refresh units stay installed, disabled, for rollback.
7 + #
8 + # Needs root on fw13: uses Tailscale SSH (node identity) to reach testnot as root
9 + # and drives the reseed there.
10 + #
11 + # Place at /etc/systemd/system/mnw-testnot-seed.service on the Sando host.
12 +
13 + [Unit]
14 + Description=MNW: reset testnot staging to the example catalog (--seed-examples)
15 + After=network-online.target
16 + Wants=network-online.target
17 +
18 + [Service]
19 + Type=oneshot
20 + ExecStart=/usr/local/bin/mnw-testnot-seed.sh
A sando/deploy/mnw-testnot-seed.sh +80
@@ -0,0 +1,80 @@
1 + #!/usr/bin/env bash
2 + # Reset testnot.work to the fabricated example catalog.
3 + #
4 + # This replaces the prod-restore refresh (mnw-testnot-refresh.sh). Instead of
5 + # reloading testnot's database from a production backup — which put real user
6 + # data on the staging box — this reseeds it from the in-repo `--seed-examples`
7 + # flow: a self-contained catalog of fabricated `@example.test` creators and
8 + # public-domain items. testnot then holds no prod-derived data at all.
9 + #
10 + # Runs on fw13 (the Sando host, which has tailnet root on testnot via Tailscale
11 + # SSH). It stops the app, resets the schema as the postgres superuser (streamed
12 + # over Tailscale SSH), then runs the app binary once with `--seed-examples`,
13 + # which applies migrations to the empty schema and runs the guarded seed before
14 + # exiting. The seed's own guards (ALLOW_EXAMPLE_SEED, testnot-host allowlist,
15 + # no-real-users) make it refuse anywhere but a testnot/localhost box.
16 + #
17 + # Idempotent and safe to re-run: the seed is fixed, and every run rebuilds the
18 + # same catalog. Unlike the refresh, there is no pause-flag — the catalog is
19 + # stable by construction, so a soak needs no protection from a daily wipe.
20 + #
21 + # Media (previews/downloads) attaches in-process only when object storage is
22 + # configured in the env file (S3_*). Until MinIO is stood up on testnot, items
23 + # seed hidden (scan_status stays pending) and the catalog shows creators,
24 + # projects, blog posts, and follow counts without media.
25 + set -euo pipefail
26 +
27 + SSH_TARGET="${TESTNOT_SSH:-root@testnot}"
28 + DB="${TESTNOT_DB:-makenotwork}"
29 + SERVICE="makenotwork.service"
30 + BIN="${TESTNOT_BIN:-/opt/mnw/current/makenotwork}"
31 + BIN_DIR="$(dirname "$BIN")"
32 + BIN_NAME="$(basename "$BIN")"
33 + ENV_FILE="${TESTNOT_ENV:-/etc/mnw/makenotwork.env}"
34 + SEED_USER="${TESTNOT_SEED_USER:-deploy}"
35 +
36 + log() { echo "[$(date -u +%H:%M:%S)] $*"; }
37 + ts_ssh() { tailscale ssh "$SSH_TARGET" "$@"; }
38 +
39 + log "stopping $SERVICE on testnot"
40 + ts_ssh "systemctl stop $SERVICE"
41 +
42 + # Drop every non-system schema (migrations create custom schemas like
43 + # tower_sessions that survive DROP SCHEMA public CASCADE). Recreate public OWNED
44 + # BY the app role: on PG15+ a postgres-owned public grants no CREATE to other
45 + # roles, so boot migrations would fail with "no schema has been selected to
46 + # create in". This is the same reset the prod-refresh used, minus the restore.
47 + log "resetting schema"
48 + ts_ssh "sudo -u postgres psql -v ON_ERROR_STOP=1 -d $DB" <<SQL
49 + DO \$\$
50 + DECLARE s text;
51 + BEGIN
52 + FOR s IN
53 + SELECT nspname FROM pg_namespace
54 + WHERE nspname NOT LIKE 'pg_%' AND nspname <> 'information_schema'
55 + LOOP
56 + EXECUTE format('DROP SCHEMA IF EXISTS %I CASCADE', s);
57 + END LOOP;
58 + EXECUTE 'CREATE SCHEMA public AUTHORIZATION $DB';
59 + END \$\$;
60 + SQL
61 +
62 + # Run the binary as the app user with the service env sourced and the seed
63 + # opt-in flag set. `--seed-examples` applies migrations to the empty schema,
64 + # runs the guarded seed, and exits(0) before the server binds a port — so this
65 + # is safe to run while the service is stopped.
66 + log "migrating + seeding the example catalog"
67 + ts_ssh "cd $BIN_DIR && sudo -u $SEED_USER env ALLOW_EXAMPLE_SEED=1 \
68 + bash -c 'set -a; . $ENV_FILE; set +a; exec ./$BIN_NAME --seed-examples'"
69 +
70 + log "starting $SERVICE"
71 + ts_ssh "systemctl start $SERVICE"
72 +
73 + # Boot smoke: the app must come back healthy after the reseed.
74 + for _ in $(seq 1 20); do
75 + code=$(ts_ssh "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/health" || echo 000)
76 + [ "$code" = "200" ] && { log "health OK"; exit 0; }
77 + sleep 3
78 + done
79 + echo "testnot did not return healthy after reseed" >&2
80 + exit 1