| 1 |
# Sando deploy units |
| 2 |
|
| 3 |
systemd units and scripts that run on the Sando host (fw13) and on deploy |
| 4 |
targets. Host-specific secrets, certs, and tailnet IPs are **not** here — they |
| 5 |
live in the Syncthing private layer (`_private/infra/`, `_private/deploy`). |
| 6 |
|
| 7 |
## Files |
| 8 |
|
| 9 |
|
| 10 |
|
| 11 |
| `sandod.service` | Sando host | The Sando daemon (`sandod`). | |
| 12 |
| `bootstrap-sandod-host.sh` | Sando host | One-time host setup for the daemon. | |
| 13 |
| `bootstrap-node.sh` | a deploy target | One-time node setup (release dirs, deploy user, service). | |
| 14 |
| `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando-daemon.toml`). Host-specific; not deployed from the repo. | |
| 15 |
| `post-receive` | git remote | Push-to-deploy hook. | |
| 16 |
| `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of every configured prod dump into `/srv/sando/backups/` (04:00 UTC). | |
| 17 |
| `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. Ends by running the smoke check below. | |
| 18 |
| `mnw-testnot-smoke.sh` | anywhere with network | Fail loudly when the testnot demo goes thin: storefronts that render fewer items than discover advertises, covers that 404 behind a 200 page, empty feeds. Checks content, not status codes. | |
| 19 |
| `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. | |
| 20 |
| `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. | |
| 21 |
| `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). | |
| 22 |
| `check-topology-drift.sh` | Sando host | Diff `main:sando/sando.toml` against the live `/etc/sando/sando.toml`; non-zero on drift. | |
| 23 |
| `sando-config-drift.{service,timer}` | Sando host | Daily run of the check above (05:00 UTC). A failed unit is the drift signal. | |
| 24 |
| `mnw-companion.sudoers` | a deploy target | Scoped sudo grant for the deploy user to run `install-companion.sh`. | |
| 25 |
|
| 26 |
## The topology is deployed, not hand-maintained |
| 27 |
|
| 28 |
`sando.toml` (the topology: tiers, nodes, gates, companions) is installed from |
| 29 |
the repo. Bootstrap installs it, and `sando-self-update.sh` reinstalls it from |
| 30 |
the checked-out sha on every self-update, so the way to change what `sandod` |
| 31 |
reads is to edit `MNW/sando/sando.toml` and ship it. |
| 32 |
|
| 33 |
This is not how it started. The repo copy read as a source of truth and was |
| 34 |
deployed nowhere, `/etc/sando/sando.toml` was hand-edited, and nothing kept the |
| 35 |
two in step: the multithreaded companion block drifted for 19 days and was found |
| 36 |
by accident while diffing before an unrelated edit. |
| 37 |
|
| 38 |
Three things hold it together now: |
| 39 |
|
| 40 |
- The self-update installs the topology **before** its `--check-config` |
| 41 |
self-test, so the test validates the binary and topology that will boot |
| 42 |
together. A failed test or a failed restart restores the previous topology, so |
| 43 |
a refused self-update leaves the box as it found it. The replaced copy is kept |
| 44 |
as `sando.toml.bak-<timestamp>`. |
| 45 |
- `sando-config-drift.timer` reports the gap that opens between self-updates: a |
| 46 |
commit on main nobody has shipped yet, or a hand edit to the live file. It |
| 47 |
reads and reports, never installs. |
| 48 |
- The daemon config beside it, `sando-daemon.toml`, stays host-specific and |
| 49 |
hand-held: bind address, tokens, database URLs. `sando-daemon.toml.example` is |
| 50 |
a template, not a deployable file. Secrets live in `sando.env` (0640 |
| 51 |
root:sando) and are untouched by any of this. |
| 52 |
|
| 53 |
`sandod` does none of the installing itself. It cannot write `/etc/sando` under |
| 54 |
`ProtectSystem=strict`, and replacing the config it is running on from inside |
| 55 |
itself would be a bootstrap loop regardless. |
| 56 |
|
| 57 |
## Self-update (deploying the controller itself) |
| 58 |
|
| 59 |
The deploy pipeline ships the *MNW server*, not `sandod`. To deploy a new |
| 60 |
`sandod` (the controller's own code), `sandod` exposes `POST /self-update |
| 61 |
{"sha":"<hex>"}` (bearer-gated like the other mutators). It cannot do the work |
| 62 |
itself — it runs `User=sando`, `NoNewPrivileges`, `ProtectSystem=strict`, so it |
| 63 |
can neither write `/usr/local/bin/sandod` nor restart its own service. It only |
| 64 |
*triggers* the root oneshot `sando-update@<sha>.service`, which the `sando` user |
| 65 |
is allowed to `start` (and nothing else) by `10-sando-update.rules`. That unit |
| 66 |
runs `sando-self-update.sh`: it builds `sando/daemon` **as the sando user** in a |
| 67 |
dedicated checkout (`/srv/sando/self-update`, never the operator's dev tree), |
| 68 |
then — as root — installs the binary and `systemctl restart sandod`. |
| 69 |
|
| 70 |
Three hardening properties gate that install (postmortem 2026-07-09 recovery): |
| 71 |
|
| 72 |
- **Build source = the local bare repo** `/srv/sando/mnw.git`, never a remote |
| 73 |
fetch by the build. Building from `git@ssh.makenot.work` gave the `sando` user |
| 74 |
no git creds and broke the moment git hosting was down (postmortem #7). |
| 75 |
`sandod` itself fetches the deploy branch into that bare repo at the top of the |
| 76 |
`/self-update` handler, so a sha pushed minutes ago is reachable. Until 0.2.9 |
| 77 |
the repo advanced only when a *server* build fetched during `/rebuild`, which |
| 78 |
chained the controller's currency to the server's release cadence and made a |
| 79 |
controller fix unshippable while the server was red. `sandod` remains the only |
| 80 |
writer of that repo, so the provenance seal below is unchanged. |
| 81 |
- **Clean build** (`cargo clean` first) so no stale incremental object survives |
| 82 |
across shas. A reused pre-`node_health` `Gate` enum object once produced a |
| 83 |
binary that crash-looped on the current config. |
| 84 |
- **`--check-config` self-test before the swap.** The freshly built binary must |
| 85 |
load + parse the live daemon config + topology (`sandod --check-config`, run as |
| 86 |
the sando user against `/etc/sando/sando-daemon.toml`). A binary that can't |
| 87 |
understand the current config fails here and is never installed — `sandod` |
| 88 |
keeps running on the old one instead of crash-looping on the new. |
| 89 |
|
| 90 |
Trigger `/self-update` only when no MNW build/deploy is in flight — `sandod` |
| 91 |
rejects it in that window anyway (the restart would SIGKILL an in-flight deploy), |
| 92 |
but treat controller updates as a separate, deliberate op. |
| 93 |
|
| 94 |
Verify a self-update landed by polling `/state`: the new `sandod_version` field |
| 95 |
reports the running controller's package version (distinct from the tier |
| 96 |
versions, which are the deployed product). |
| 97 |
|
| 98 |
Do **not** poll `systemctl is-active sando-update@<sha>` for that. The unit |
| 99 |
reports `activating` for the whole build, so `until ! systemctl is-active ...` |
| 100 |
exits immediately and reads a running build as a finished one. Poll |
| 101 |
`ActiveState` explicitly, or poll `/state` for the `sandod_version` you expect. |
| 102 |
|
| 103 |
One-time install (as root), then every future controller deploy is one API call: |
| 104 |
|
| 105 |
```sh |
| 106 |
sudo install -d /usr/local/lib/sando |
| 107 |
sudo install -m 0755 sando-self-update.sh /usr/local/lib/sando/sando-self-update.sh |
| 108 |
sudo install -m 0644 sando-update@.service /etc/systemd/system/ |
| 109 |
sudo install -m 0644 10-sando-update.rules /etc/polkit-1/rules.d/ |
| 110 |
sudo systemctl daemon-reload |
| 111 |
``` |
| 112 |
|
| 113 |
Bootstrap caveat: the *first* `sandod` carrying `/self-update` still has to be |
| 114 |
installed by hand (build + `install` + `systemctl restart sandod`) — there is no |
| 115 |
endpoint to call until it is running. Self-update is for every deploy after that. |
| 116 |
Trigger + watch: |
| 117 |
|
| 118 |
```sh |
| 119 |
# The sha must be on the canonical remote's main — that is what sandod fetches |
| 120 |
# and what the updater's ancestry check seals against. Read it from the remote, |
| 121 |
# not from the working copy, which may be ahead of what was pushed. |
| 122 |
SHA=$(git -C ~/Code/MNW ls-remote mnw main | cut -f1) |
| 123 |
curl -sS -X POST "$BASE/self-update" -H 'Content-Type: application/json' \ |
| 124 |
-H "Authorization: Bearer $SANDO_API_TOKEN" -d "{\"sha\":\"$SHA\"}" |
| 125 |
journalctl -u "sando-update@$SHA" -f |
| 126 |
``` |
| 127 |
|
| 128 |
## Editing a node's env file |
| 129 |
|
| 130 |
`/etc/mnw/makenotwork.env` must stay **mode 0640, owned root:<service user>**. |
| 131 |
`bootstrap-node.sh` creates it that way and the deploy depends on it. |
| 132 |
|
| 133 |
The trap, which cost a failed prod deploy on 2026-08-01: systemd reads |
| 134 |
`EnvironmentFile=` as root before dropping to `User=`, so the running service |
| 135 |
does not care about the mode. Sando's pre-swap config check is the only thing |
| 136 |
that reads the file **as the deploy user**, because its whole job is to load the |
| 137 |
config the way the service will before swapping the symlink. So a file rewritten |
| 138 |
0600 leaves production serving happily while the next deploy is already broken, |
| 139 |
and nothing says so until someone ships. |
| 140 |
|
| 141 |
Rewriting the file through a temp file is what does it: |
| 142 |
|
| 143 |
# WRONG: the temp file carries root's 077 umask, and `mv` carries it over |
| 144 |
{ cat /etc/mnw/makenotwork.env; echo "NEW_VAR=x"; } > /tmp/env && \ |
| 145 |
mv /tmp/env /etc/mnw/makenotwork.env |
| 146 |
|
| 147 |
`mv` replaces the inode, so the new file keeps the temp's 0600 and loses any ACL |
| 148 |
the old one carried. Append in place, or restore the mode explicitly afterwards: |
| 149 |
|
| 150 |
# RIGHT: in-place, mode preserved |
| 151 |
echo "NEW_VAR=x" >> /etc/mnw/makenotwork.env |
| 152 |
|
| 153 |
# or, if the file must be rewritten |
| 154 |
install -m 0640 -o root -g "$SERVICE_USER" /tmp/env /etc/mnw/makenotwork.env |
| 155 |
|
| 156 |
The deploy now fails with a readability probe naming the user, the mode and the |
| 157 |
owner rather than a bare `Permission denied` from inside a generated script, so |
| 158 |
this is recoverable in one read — but it is still a failed deploy. |
| 159 |
|
| 160 |
## Rollback contract |
| 161 |
|
| 162 |
A Sando rollback (canary rollback of a node, or an operator swapping the |
| 163 |
`current` symlink back to an older release dir) restores **the binary and the |
| 164 |
release contents only**. The database does not roll back: |
| 165 |
|
| 166 |
- MNW migrates **forward** on boot (`makenotwork.service` runs pending migrations |
| 167 |
when it starts). |
| 168 |
- `sqlx::migrate::Migrator` has no down path, and Sando never invokes one. |
| 169 |
|
| 170 |
So rolling, say, `0.10.15` back to `0.10.14` **after** `0.10.15` applied a |
| 171 |
migration leaves the old binary running against a newer schema. That is a |
| 172 |
one-way door for any release that carries a migration: forward is safe, back is |
| 173 |
not. Sando has no restore-to-prod path either — `/backup/fetch` pulls the prod |
| 174 |
dump solely as input to the `migration_dry_run` gate, not to restore a live node. |
| 175 |
|
| 176 |
**Operating rule.** Promote a migration-bearing release with |
| 177 |
`{"bears_migration": true}`. That forces a fresh `manual_confirm` on the |
| 178 |
predecessor tier before the advance, even on a tier that configures no confirm, |
| 179 |
so the one-way advance is a conscious step. `hotfix` does not suppress it (it |
| 180 |
skips only `burn_in`). To actually undo a migration-bearing release you must |
| 181 |
restore the database from a backup by hand first; a symlink rollback alone will |
| 182 |
serve a mismatched schema. |
| 183 |
|
| 184 |
**Does the backup actually restore?** The `migration_dry_run` gate answers this |
| 185 |
on every build, once per `[[migration_check]]`: it resets that check's scratch |
| 186 |
database, restores the latest dump of the database the check is for, and runs |
| 187 |
the migrator against it. A failed restore fails the gate. Both prod databases |
| 188 |
are covered — `makenotwork` and `multithreaded`, each from its own dump under |
| 189 |
`/srv/sando/backups/`. A check that names a scratch database of its own gets it |
| 190 |
created by the daemon on first run, so adding one owes no step here. What is |
| 191 |
*not* automated is a full restore-to-serving drill (restore into a throwaway |
| 192 |
target and confirm the app boots and serves |
| 193 |
against it) — that is tracked as an infra task, not wired into the pipeline. |
| 194 |
|
| 195 |
## Companion services (deploying mnw-cli in lockstep) |
| 196 |
|
| 197 |
`mnw-cli` (the public git-SSH server that proxies to `/api/internal/*`) shares |
| 198 |
the server's internal-API contract but used to deploy from its own |
| 199 |
`mnw-cli/deploy/deploy.sh`. It drifted two months out of lockstep and broke git |
| 200 |
hosting the moment the server tightened that contract (0.10.14, postmortem #4). |
| 201 |
Sando now builds and ships it in the **same promote** as the server: |
| 202 |
|
| 203 |
- **Build** — `[[companion]]` in the daemon config lists crates to compile from |
| 204 |
the same worktree/sha as the server. Each is built after the server and staged |
| 205 |
into the release bundle as `companions/<name>`. A companion that fails to build |
| 206 |
fails the whole pipeline — that is the lockstep guarantee. |
| 207 |
- **Deploy** — `[[tier.node.companion]]` on a node says which companions it |
| 208 |
installs, to what `install_path`, and which unit to restart. After the server |
| 209 |
is swapped and back up (mnw-cli is `After=makenotwork.service`), the node |
| 210 |
installs the staged binary and restarts the unit. testnot has no mnw-cli, so |
| 211 |
only `prod-1` declares it. |
| 212 |
|
| 213 |
Deploy runs `sudo /usr/local/lib/mnw/install-companion.sh <src> <dst> <service>` |
| 214 |
over the node's executor. The wrapper keeps the deploy user's sudo grant to one |
| 215 |
auditable script (it validates: src inside a release bundle, dst under `/opt`, |
| 216 |
service a bare `*.service`) rather than a broad `install`/`systemctl` grant. |
| 217 |
|
| 218 |
The grant names every deploy user Sando SSHes as — `makenotwork` for prod-1, |
| 219 |
`deploy` for testnot-1 — so the file installs verbatim on any node and needs no |
| 220 |
hand-edit at install time. sudoers accepts a user that is not present on the box, |
| 221 |
so the other node's line is inert rather than an error. A node added later needs |
| 222 |
its `ssh_target` user added to `mnw-companion.sudoers`. |
| 223 |
|
| 224 |
One-time per node that hosts a companion (currently prod-1), as root: |
| 225 |
|
| 226 |
```sh |
| 227 |
sudo install -d /usr/local/lib/mnw |
| 228 |
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh |
| 229 |
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion |
| 230 |
sudo visudo -cf /etc/sudoers.d/mnw-companion # validate before trusting it |
| 231 |
``` |
| 232 |
|
| 233 |
After that, every promote that reaches the node ships the server and its |
| 234 |
companions together; no separate `mnw-cli` deploy step. The legacy |
| 235 |
`mnw-cli/deploy/deploy.sh` is retired once the first lockstep prod promote lands. |
| 236 |
|
| 237 |
## testnot.work staging |
| 238 |
|
| 239 |
testnot is gated app-side to Fan+/creator accounts |
| 240 |
(`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+ members can |
| 241 |
preview upcoming features, and to back the pre-cutover migration dry-run. |
| 242 |
|
| 243 |
**Source of truth: the seeded example catalog** (`mnw-testnot-seed.sh`). testnot |
| 244 |
no longer mirrors production. Instead it runs a self-contained catalog of |
| 245 |
fabricated `@example.test` creators and public-domain items, built by the app's |
| 246 |
`--seed-examples` flow — so no prod-derived data lives on the staging box. The |
| 247 |
script stops the app, resets the schema (recreating `public` owned by the app |
| 248 |
role — PG15+ otherwise blocks the app role's boot migrations), then runs the |
| 249 |
binary once with `--seed-examples`, which migrates the empty schema and runs the |
| 250 |
guarded seed before exiting, and restarts the app. It is idempotent — re-run it |
| 251 |
any time to reset testnot to the fixed catalog. There is no pause-flag; the |
| 252 |
catalog is stable by construction. |
| 253 |
|
| 254 |
Media (previews/downloads) attaches only when object storage is configured in |
| 255 |
`/etc/mnw/makenotwork.env` (`S3_*`, plus `S3_PUBLIC_BUCKET` + `CDN_BASE_URL` for |
| 256 |
covers). Until MinIO is stood up on testnot, items seed **hidden** (creators, |
| 257 |
projects, blog posts, and follow counts still show). |
| 258 |
|
| 259 |
Install on the Sando host: |
| 260 |
|
| 261 |
```sh |
| 262 |
sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh |
| 263 |
sudo install -m 0755 mnw-testnot-smoke.sh /usr/local/bin/mnw-testnot-smoke.sh |
| 264 |
sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/ |
| 265 |
sudo systemctl daemon-reload |
| 266 |
``` |
| 267 |
|
| 268 |
Install both. The seed script ends by `exec`ing the smoke check from its own |
| 269 |
directory, so a seed installed without its sibling reseeds and then fails at the |
| 270 |
last step. |
| 271 |
|
| 272 |
Reset testnot any time: `sudo /usr/local/bin/mnw-testnot-seed.sh` |
| 273 |
(or `sudo systemctl start mnw-testnot-seed.service`). |
| 274 |
|
| 275 |
A reseed is not finished when the health check passes. Healthy is not the same |
| 276 |
as worth showing: every page can return 200 while the catalog is empty, which is |
| 277 |
how nine of eleven items stayed invisible for weeks. The smoke check is the part |
| 278 |
that notices, and it runs automatically at the end of every seed. Run it alone |
| 279 |
whenever you want the same answer: `./mnw-testnot-smoke.sh`. Set |
| 280 |
`SKIP_CONTENT_SMOKE=1` on the seed to skip it while the catalog is deliberately |
| 281 |
mid-change. |
| 282 |
|
| 283 |
Its floors are sealed baselines in the same spirit as `tests/test_hygiene.rs`: |
| 284 |
they freeze what is true today and fail on a new violation. Move one only in the |
| 285 |
improving direction, and never loosen one to make a run pass. |
| 286 |
|
| 287 |
The seed flow is the only way testnot gets its data. The prod-restore refresh it |
| 288 |
replaced was deleted on 2026-07-22; testnot holds no prod-derived data and there |
| 289 |
is no path that puts any there. |
| 290 |
|
| 291 |
**Redeploy the binary** (no Sando integration yet — manual): build the release |
| 292 |
on the Sando host, then over Tailscale SSH as root on the target, copy the |
| 293 |
current release dir to a new one, stream the new `makenotwork` binary in, stream |
| 294 |
a tar of `static/` + `docs/` (`docs/` = `server/site-docs/{public,examples}` + |
| 295 |
`server/docs/business/assumptions.toml`, mirroring |
| 296 |
`server/deploy/archive/deploy.sh.legacy`), |
| 297 |
flip the `current` symlink, and `systemctl restart makenotwork.service` (which |
| 298 |
boot-migrates). Old release dirs are kept for rollback. |
| 299 |
|
| 300 |
Cert, the makenotwork Postgres password, the Caddyfile, and the operator |
| 301 |
runbook for the root-level node setup are in `_private/infra/testnot/`. |
| 302 |
|