# Sando deploy units

systemd units and scripts that run on the Sando host (fw13) and on deploy
targets. Host-specific secrets, certs, and tailnet IPs are **not** here — they
live in the Syncthing private layer (`_private/infra/`, `_private/deploy`).

## Files

| File | Where it runs | Purpose |
|------|---------------|---------|
| `sandod.service` | Sando host | The Sando daemon (`sandod`). |
| `bootstrap-sandod-host.sh` | Sando host | One-time host setup for the daemon. |
| `bootstrap-node.sh` | a deploy target | One-time node setup (release dirs, deploy user, service). |
| `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando.toml`). |
| `post-receive` | git remote | Push-to-deploy hook. |
| `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of the prod backup to `/srv/sando/backups/latest.sql.gz` (04:00 UTC). |
| `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. |
| `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. |
| `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. |
| `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). |
| `mnw-companion.sudoers` | a deploy target | Scoped sudo grant for the deploy user to run `install-companion.sh`. |

## Self-update (deploying the controller itself)

The deploy pipeline ships the *MNW server*, not `sandod`. To deploy a new
`sandod` (the controller's own code), `sandod` exposes `POST /self-update
{"sha":"<hex>"}` (bearer-gated like the other mutators). It cannot do the work
itself — it runs `User=sando`, `NoNewPrivileges`, `ProtectSystem=strict`, so it
can neither write `/usr/local/bin/sandod` nor restart its own service. It only
*triggers* the root oneshot `sando-update@<sha>.service`, which the `sando` user
is allowed to `start` (and nothing else) by `10-sando-update.rules`. That unit
runs `sando-self-update.sh`: it builds `sando/daemon` **as the sando user** in a
dedicated checkout (`/srv/sando/self-update`, never the operator's dev tree),
then — as root — installs the binary and `systemctl restart sandod`.

Three hardening properties gate that install (postmortem 2026-07-09 recovery):

- **Source = the local bare repo** `/srv/sando/mnw.git`, not a remote fetch. The
  self-update sha was already deployed as a server release, so `/rebuild` has
  force-updated the bare repo's `main` to include it — the sha and a trusted
  provenance ref are both present locally. A remote fetch gave the `sando` user
  no git creds and broke the moment git hosting was down.
- **Clean build** (`cargo clean` first) so no stale incremental object survives
  across shas. A reused pre-`node_health` `Gate` enum object once produced a
  binary that crash-looped on the current config.
- **`--check-config` self-test before the swap.** The freshly built binary must
  load + parse the live daemon config + topology (`sandod --check-config`, run as
  the sando user against `/etc/sando/sando-daemon.toml`). A binary that can't
  understand the current config fails here and is never installed — `sandod`
  keeps running on the old one instead of crash-looping on the new.

Trigger `/self-update` only when no MNW build/deploy is in flight — `sandod`
rejects it in that window anyway (the restart would SIGKILL an in-flight deploy),
but treat controller updates as a separate, deliberate op.

Verify a self-update landed by polling `/state`: the new `sandod_version` field
reports the running controller's package version (distinct from the tier
versions, which are the deployed product).

One-time install (as root), then every future controller deploy is one API call:

```sh
sudo install -d /usr/local/lib/sando
sudo install -m 0755 sando-self-update.sh /usr/local/lib/sando/sando-self-update.sh
sudo install -m 0644 sando-update@.service /etc/systemd/system/
sudo install -m 0644 10-sando-update.rules /etc/polkit-1/rules.d/
sudo systemctl daemon-reload
```

Bootstrap caveat: the *first* `sandod` carrying `/self-update` still has to be
installed by hand (build + `install` + `systemctl restart sandod`) — there is no
endpoint to call until it is running. Self-update is for every deploy after that.
Trigger + watch:

```sh
SHA=$(git -C ~/Code/MNW rev-parse HEAD)
curl -sS -X POST "$BASE/self-update" -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $SANDO_API_TOKEN" -d "{\"sha\":\"$SHA\"}"
journalctl -u "sando-update@$SHA" -f
```

## Rollback contract

A Sando rollback (canary rollback of a node, or an operator swapping the
`current` symlink back to an older release dir) restores **the binary and the
release contents only**. The database does not roll back:

- MNW migrates **forward** on boot (`makenotwork.service` runs pending migrations
  when it starts).
- `sqlx::migrate::Migrator` has no down path, and Sando never invokes one.

So rolling, say, `0.10.15` back to `0.10.14` **after** `0.10.15` applied a
migration leaves the old binary running against a newer schema. That is a
one-way door for any release that carries a migration: forward is safe, back is
not. Sando has no restore-to-prod path either — `/backup/fetch` pulls the prod
dump solely as input to the `migration_dry_run` gate, not to restore a live node.

**Operating rule.** Promote a migration-bearing release with
`{"bears_migration": true}`. That forces a fresh `manual_confirm` on the
predecessor tier before the advance, even on a tier that configures no confirm,
so the one-way advance is a conscious step. `hotfix` does not suppress it (it
skips only `burn_in`). To actually undo a migration-bearing release you must
restore the database from a backup by hand first; a symlink rollback alone will
serve a mismatched schema.

**Does the backup actually restore?** The `migration_dry_run` gate answers this
on every build: it resets a scratch database, restores the latest
`/srv/sando/backups/latest.sql.gz`, and runs the migrator against it. A failed
restore fails the gate. What is *not* automated is a full restore-to-serving
drill (restore into a throwaway target and confirm the app boots and serves
against it) — that is tracked as an infra task, not wired into the pipeline.

## Companion services (deploying mnw-cli in lockstep)

`mnw-cli` (the public git-SSH server that proxies to `/api/internal/*`) shares
the server's internal-API contract but used to deploy from its own
`mnw-cli/deploy/deploy.sh`. It drifted two months out of lockstep and broke git
hosting the moment the server tightened that contract (0.10.14, postmortem #4).
Sando now builds and ships it in the **same promote** as the server:

- **Build** — `[[companion]]` in the daemon config lists crates to compile from
  the same worktree/sha as the server. Each is built after the server and staged
  into the release bundle as `companions/<name>`. A companion that fails to build
  fails the whole pipeline — that is the lockstep guarantee.
- **Deploy** — `[[tier.node.companion]]` on a node says which companions it
  installs, to what `install_path`, and which unit to restart. After the server
  is swapped and back up (mnw-cli is `After=makenotwork.service`), the node
  installs the staged binary and restarts the unit. testnot has no mnw-cli, so
  only `prod-1` declares it.

Deploy runs `sudo /usr/local/lib/mnw/install-companion.sh <src> <dst> <service>`
over the node's executor. The wrapper keeps the deploy user's sudo grant to one
auditable script (it validates: src inside a release bundle, dst under `/opt`,
service a bare `*.service`) rather than a broad `install`/`systemctl` grant.

One-time per node that hosts a companion (currently prod-1), as root:

```sh
sudo install -d /usr/local/lib/mnw
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion
sudo visudo -cf /etc/sudoers.d/mnw-companion   # validate before trusting it
```

After that, every promote that reaches the node ships the server and its
companions together; no separate `mnw-cli` deploy step. The legacy
`mnw-cli/deploy/deploy.sh` is retired once the first lockstep prod promote lands.

## testnot.work staging

testnot is gated app-side to Fan+/creator accounts
(`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+ members can
preview upcoming features, and to back the pre-cutover migration dry-run.

**Source of truth: the seeded example catalog** (`mnw-testnot-seed.sh`). testnot
no longer mirrors production. Instead it runs a self-contained catalog of
fabricated `@example.test` creators and public-domain items, built by the app's
`--seed-examples` flow — so no prod-derived data lives on the staging box. The
script stops the app, resets the schema (recreating `public` owned by the app
role — PG15+ otherwise blocks the app role's boot migrations), then runs the
binary once with `--seed-examples`, which migrates the empty schema and runs the
guarded seed before exiting, and restarts the app. It is idempotent — re-run it
any time to reset testnot to the fixed catalog. There is no pause-flag; the
catalog is stable by construction.

Media (previews/downloads) attaches only when object storage is configured in
`/etc/mnw/makenotwork.env` (`S3_*`, plus `S3_PUBLIC_BUCKET` + `CDN_BASE_URL` for
covers). Until MinIO is stood up on testnot, items seed **hidden** (creators,
projects, blog posts, and follow counts still show).

Install on the Sando host:

```sh
sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh
sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/
sudo systemctl daemon-reload
```

Reset testnot any time: `sudo /usr/local/bin/mnw-testnot-seed.sh`
(or `sudo systemctl start mnw-testnot-seed.service`).

The seed flow is the only way testnot gets its data. The prod-restore refresh it
replaced was deleted on 2026-07-22; testnot holds no prod-derived data and there
is no path that puts any there.

**Redeploy the binary** (no Sando integration yet — manual): build the release
on the Sando host, then over Tailscale SSH as root on the target, copy the
current release dir to a new one, stream the new `makenotwork` binary in, stream
a tar of `static/` + `docs/` (`docs/` = `server/site-docs/{public,examples}` +
`server/docs/business/assumptions.toml`, mirroring `server/deploy/deploy.sh`),
flip the `current` symlink, and `systemctl restart makenotwork.service` (which
boot-migrates). Old release dirs are kept for rollback.

Cert, the makenotwork Postgres password, the Caddyfile, and the operator
runbook for the root-level node setup are in `_private/infra/testnot/`.