| 1 |
# Sando deploy units |
| 2 |
|
| 3 |
systemd units and scripts that run on the Sando host (fw13) and on deploy |
| 4 |
targets. Host-specific secrets, certs, and tailnet IPs are **not** here — they |
| 5 |
live in the Syncthing private layer (`_private/infra/`, `_private/deploy`). |
| 6 |
|
| 7 |
## Files |
| 8 |
|
| 9 |
|
| 10 |
|
| 11 |
| `sandod.service` | Sando host | The Sando daemon (`sandod`). | |
| 12 |
| `bootstrap-sandod-host.sh` | Sando host | One-time host setup for the daemon. | |
| 13 |
| `bootstrap-node.sh` | a deploy target | One-time node setup (release dirs, deploy user, service). | |
| 14 |
| `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando.toml`). | |
| 15 |
| `post-receive` | git remote | Push-to-deploy hook. | |
| 16 |
| `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of the prod backup to `/srv/sando/backups/latest.sql.gz` (04:00 UTC). | |
| 17 |
| `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. | |
| 18 |
| `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. | |
| 19 |
| `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. | |
| 20 |
| `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). | |
| 21 |
| `mnw-companion.sudoers` | a deploy target | Scoped sudo grant for the deploy user to run `install-companion.sh`. | |
| 22 |
|
| 23 |
## Self-update (deploying the controller itself) |
| 24 |
|
| 25 |
The deploy pipeline ships the *MNW server*, not `sandod`. To deploy a new |
| 26 |
`sandod` (the controller's own code), `sandod` exposes `POST /self-update |
| 27 |
{"sha":"<hex>"}` (bearer-gated like the other mutators). It cannot do the work |
| 28 |
itself — it runs `User=sando`, `NoNewPrivileges`, `ProtectSystem=strict`, so it |
| 29 |
can neither write `/usr/local/bin/sandod` nor restart its own service. It only |
| 30 |
*triggers* the root oneshot `sando-update@<sha>.service`, which the `sando` user |
| 31 |
is allowed to `start` (and nothing else) by `10-sando-update.rules`. That unit |
| 32 |
runs `sando-self-update.sh`: it builds `sando/daemon` **as the sando user** in a |
| 33 |
dedicated checkout (`/srv/sando/self-update`, never the operator's dev tree), |
| 34 |
then — as root — installs the binary and `systemctl restart sandod`. |
| 35 |
|
| 36 |
Three hardening properties gate that install (postmortem 2026-07-09 recovery): |
| 37 |
|
| 38 |
- **Source = the local bare repo** `/srv/sando/mnw.git`, not a remote fetch. The |
| 39 |
self-update sha was already deployed as a server release, so `/rebuild` has |
| 40 |
force-updated the bare repo's `main` to include it — the sha and a trusted |
| 41 |
provenance ref are both present locally. A remote fetch gave the `sando` user |
| 42 |
no git creds and broke the moment git hosting was down. |
| 43 |
- **Clean build** (`cargo clean` first) so no stale incremental object survives |
| 44 |
across shas. A reused pre-`node_health` `Gate` enum object once produced a |
| 45 |
binary that crash-looped on the current config. |
| 46 |
- **`--check-config` self-test before the swap.** The freshly built binary must |
| 47 |
load + parse the live daemon config + topology (`sandod --check-config`, run as |
| 48 |
the sando user against `/etc/sando/sando-daemon.toml`). A binary that can't |
| 49 |
understand the current config fails here and is never installed — `sandod` |
| 50 |
keeps running on the old one instead of crash-looping on the new. |
| 51 |
|
| 52 |
Trigger `/self-update` only when no MNW build/deploy is in flight — `sandod` |
| 53 |
rejects it in that window anyway (the restart would SIGKILL an in-flight deploy), |
| 54 |
but treat controller updates as a separate, deliberate op. |
| 55 |
|
| 56 |
Verify a self-update landed by polling `/state`: the new `sandod_version` field |
| 57 |
reports the running controller's package version (distinct from the tier |
| 58 |
versions, which are the deployed product). |
| 59 |
|
| 60 |
One-time install (as root), then every future controller deploy is one API call: |
| 61 |
|
| 62 |
```sh |
| 63 |
sudo install -d /usr/local/lib/sando |
| 64 |
sudo install -m 0755 sando-self-update.sh /usr/local/lib/sando/sando-self-update.sh |
| 65 |
sudo install -m 0644 sando-update@.service /etc/systemd/system/ |
| 66 |
sudo install -m 0644 10-sando-update.rules /etc/polkit-1/rules.d/ |
| 67 |
sudo systemctl daemon-reload |
| 68 |
``` |
| 69 |
|
| 70 |
Bootstrap caveat: the *first* `sandod` carrying `/self-update` still has to be |
| 71 |
installed by hand (build + `install` + `systemctl restart sandod`) — there is no |
| 72 |
endpoint to call until it is running. Self-update is for every deploy after that. |
| 73 |
Trigger + watch: |
| 74 |
|
| 75 |
```sh |
| 76 |
SHA=$(git -C ~/Code/MNW rev-parse HEAD) |
| 77 |
curl -sS -X POST "$BASE/self-update" -H 'Content-Type: application/json' \ |
| 78 |
-H "Authorization: Bearer $SANDO_API_TOKEN" -d "{\"sha\":\"$SHA\"}" |
| 79 |
journalctl -u "sando-update@$SHA" -f |
| 80 |
``` |
| 81 |
|
| 82 |
## Rollback contract |
| 83 |
|
| 84 |
A Sando rollback (canary rollback of a node, or an operator swapping the |
| 85 |
`current` symlink back to an older release dir) restores **the binary and the |
| 86 |
release contents only**. The database does not roll back: |
| 87 |
|
| 88 |
- MNW migrates **forward** on boot (`makenotwork.service` runs pending migrations |
| 89 |
when it starts). |
| 90 |
- `sqlx::migrate::Migrator` has no down path, and Sando never invokes one. |
| 91 |
|
| 92 |
So rolling, say, `0.10.15` back to `0.10.14` **after** `0.10.15` applied a |
| 93 |
migration leaves the old binary running against a newer schema. That is a |
| 94 |
one-way door for any release that carries a migration: forward is safe, back is |
| 95 |
not. Sando has no restore-to-prod path either — `/backup/fetch` pulls the prod |
| 96 |
dump solely as input to the `migration_dry_run` gate, not to restore a live node. |
| 97 |
|
| 98 |
**Operating rule.** Promote a migration-bearing release with |
| 99 |
`{"bears_migration": true}`. That forces a fresh `manual_confirm` on the |
| 100 |
predecessor tier before the advance, even on a tier that configures no confirm, |
| 101 |
so the one-way advance is a conscious step. `hotfix` does not suppress it (it |
| 102 |
skips only `burn_in`). To actually undo a migration-bearing release you must |
| 103 |
restore the database from a backup by hand first; a symlink rollback alone will |
| 104 |
serve a mismatched schema. |
| 105 |
|
| 106 |
**Does the backup actually restore?** The `migration_dry_run` gate answers this |
| 107 |
on every build: it resets a scratch database, restores the latest |
| 108 |
`/srv/sando/backups/latest.sql.gz`, and runs the migrator against it. A failed |
| 109 |
restore fails the gate. What is *not* automated is a full restore-to-serving |
| 110 |
drill (restore into a throwaway target and confirm the app boots and serves |
| 111 |
against it) — that is tracked as an infra task, not wired into the pipeline. |
| 112 |
|
| 113 |
## Companion services (deploying mnw-cli in lockstep) |
| 114 |
|
| 115 |
`mnw-cli` (the public git-SSH server that proxies to `/api/internal/*`) shares |
| 116 |
the server's internal-API contract but used to deploy from its own |
| 117 |
`mnw-cli/deploy/deploy.sh`. It drifted two months out of lockstep and broke git |
| 118 |
hosting the moment the server tightened that contract (0.10.14, postmortem #4). |
| 119 |
Sando now builds and ships it in the **same promote** as the server: |
| 120 |
|
| 121 |
- **Build** — `[[companion]]` in the daemon config lists crates to compile from |
| 122 |
the same worktree/sha as the server. Each is built after the server and staged |
| 123 |
into the release bundle as `companions/<name>`. A companion that fails to build |
| 124 |
fails the whole pipeline — that is the lockstep guarantee. |
| 125 |
- **Deploy** — `[[tier.node.companion]]` on a node says which companions it |
| 126 |
installs, to what `install_path`, and which unit to restart. After the server |
| 127 |
is swapped and back up (mnw-cli is `After=makenotwork.service`), the node |
| 128 |
installs the staged binary and restarts the unit. testnot has no mnw-cli, so |
| 129 |
only `prod-1` declares it. |
| 130 |
|
| 131 |
Deploy runs `sudo /usr/local/lib/mnw/install-companion.sh <src> <dst> <service>` |
| 132 |
over the node's executor. The wrapper keeps the deploy user's sudo grant to one |
| 133 |
auditable script (it validates: src inside a release bundle, dst under `/opt`, |
| 134 |
service a bare `*.service`) rather than a broad `install`/`systemctl` grant. |
| 135 |
|
| 136 |
One-time per node that hosts a companion (currently prod-1), as root: |
| 137 |
|
| 138 |
```sh |
| 139 |
sudo install -d /usr/local/lib/mnw |
| 140 |
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh |
| 141 |
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion |
| 142 |
sudo visudo -cf /etc/sudoers.d/mnw-companion # validate before trusting it |
| 143 |
``` |
| 144 |
|
| 145 |
After that, every promote that reaches the node ships the server and its |
| 146 |
companions together; no separate `mnw-cli` deploy step. The legacy |
| 147 |
`mnw-cli/deploy/deploy.sh` is retired once the first lockstep prod promote lands. |
| 148 |
|
| 149 |
## testnot.work staging |
| 150 |
|
| 151 |
testnot is gated app-side to Fan+/creator accounts |
| 152 |
(`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+ members can |
| 153 |
preview upcoming features, and to back the pre-cutover migration dry-run. |
| 154 |
|
| 155 |
**Source of truth: the seeded example catalog** (`mnw-testnot-seed.sh`). testnot |
| 156 |
no longer mirrors production. Instead it runs a self-contained catalog of |
| 157 |
fabricated `@example.test` creators and public-domain items, built by the app's |
| 158 |
`--seed-examples` flow — so no prod-derived data lives on the staging box. The |
| 159 |
script stops the app, resets the schema (recreating `public` owned by the app |
| 160 |
role — PG15+ otherwise blocks the app role's boot migrations), then runs the |
| 161 |
binary once with `--seed-examples`, which migrates the empty schema and runs the |
| 162 |
guarded seed before exiting, and restarts the app. It is idempotent — re-run it |
| 163 |
any time to reset testnot to the fixed catalog. There is no pause-flag; the |
| 164 |
catalog is stable by construction. |
| 165 |
|
| 166 |
Media (previews/downloads) attaches only when object storage is configured in |
| 167 |
`/etc/mnw/makenotwork.env` (`S3_*`, plus `S3_PUBLIC_BUCKET` + `CDN_BASE_URL` for |
| 168 |
covers). Until MinIO is stood up on testnot, items seed **hidden** (creators, |
| 169 |
projects, blog posts, and follow counts still show). |
| 170 |
|
| 171 |
Install on the Sando host: |
| 172 |
|
| 173 |
```sh |
| 174 |
sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh |
| 175 |
sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/ |
| 176 |
sudo systemctl daemon-reload |
| 177 |
``` |
| 178 |
|
| 179 |
Reset testnot any time: `sudo /usr/local/bin/mnw-testnot-seed.sh` |
| 180 |
(or `sudo systemctl start mnw-testnot-seed.service`). |
| 181 |
|
| 182 |
The seed flow is the only way testnot gets its data. The prod-restore refresh it |
| 183 |
replaced was deleted on 2026-07-22; testnot holds no prod-derived data and there |
| 184 |
is no path that puts any there. |
| 185 |
|
| 186 |
**Redeploy the binary** (no Sando integration yet — manual): build the release |
| 187 |
on the Sando host, then over Tailscale SSH as root on the target, copy the |
| 188 |
current release dir to a new one, stream the new `makenotwork` binary in, stream |
| 189 |
a tar of `static/` + `docs/` (`docs/` = `server/site-docs/{public,examples}` + |
| 190 |
`server/docs/business/assumptions.toml`, mirroring `server/deploy/deploy.sh`), |
| 191 |
flip the `current` symlink, and `systemctl restart makenotwork.service` (which |
| 192 |
boot-migrates). Old release dirs are kept for rollback. |
| 193 |
|
| 194 |
Cert, the makenotwork Postgres password, the Caddyfile, and the operator |
| 195 |
runbook for the root-level node setup are in `_private/infra/testnot/`. |
| 196 |
|