# Sando deploy units

systemd units and scripts that run on the Sando host (fw13) and on deploy
targets. Host-specific secrets, certs, and tailnet IPs are **not** here — they
live in the Syncthing private layer (`_private/infra/`, `_private/deploy`).

## Files

| File | Where it runs | Purpose |
|------|---------------|---------|
| `sandod.service` | Sando host | The Sando daemon (`sandod`). |
| `bootstrap-sandod-host.sh` | Sando host | One-time host setup for the daemon. |
| `bootstrap-node.sh` | a deploy target | One-time node setup (release dirs, deploy user, service). |
| `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando-daemon.toml`). Host-specific; not deployed from the repo. |
| `post-receive` | git remote | Push-to-deploy hook. |
| `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of every configured prod dump into `/srv/sando/backups/` (04:00 UTC). |
| `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. Ends by running the smoke check below. |
| `mnw-testnot-smoke.sh` | anywhere with network | Fail loudly when the testnot demo goes thin: storefronts that render fewer items than discover advertises, covers that 404 behind a 200 page, empty feeds. Checks content, not status codes. |
| `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. |
| `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. |
| `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). |
| `check-topology-drift.sh` | Sando host | Diff `main:sando/sando.toml` against the live `/etc/sando/sando.toml`; non-zero on drift. |
| `sando-config-drift.{service,timer}` | Sando host | Daily run of the check above (05:00 UTC). A failed unit is the drift signal. |
| `mnw-companion.sudoers` | a deploy target | Scoped sudo grant for the deploy user to run `install-companion.sh`. |

## The topology is deployed, not hand-maintained

`sando.toml` (the topology: tiers, nodes, gates, companions) is installed from
the repo. Bootstrap installs it, and `sando-self-update.sh` reinstalls it from
the checked-out sha on every self-update, so the way to change what `sandod`
reads is to edit `MNW/sando/sando.toml` and ship it.

This is not how it started. The repo copy read as a source of truth and was
deployed nowhere, `/etc/sando/sando.toml` was hand-edited, and nothing kept the
two in step: the multithreaded companion block drifted for 19 days and was found
by accident while diffing before an unrelated edit.

Three things hold it together now:

- The self-update installs the topology **before** its `--check-config`
  self-test, so the test validates the binary and topology that will boot
  together. A failed test or a failed restart restores the previous topology, so
  a refused self-update leaves the box as it found it. The replaced copy is kept
  as `sando.toml.bak-<timestamp>`.
- `sando-config-drift.timer` reports the gap that opens between self-updates: a
  commit on main nobody has shipped yet, or a hand edit to the live file. It
  reads and reports, never installs.
- The daemon config beside it, `sando-daemon.toml`, stays host-specific and
  hand-held: bind address, tokens, database URLs. `sando-daemon.toml.example` is
  a template, not a deployable file. Secrets live in `sando.env` (0640
  root:sando) and are untouched by any of this.

`sandod` does none of the installing itself. It cannot write `/etc/sando` under
`ProtectSystem=strict`, and replacing the config it is running on from inside
itself would be a bootstrap loop regardless.

## Self-update (deploying the controller itself)

The deploy pipeline ships the *MNW server*, not `sandod`. To deploy a new
`sandod` (the controller's own code), `sandod` exposes `POST /self-update
{"sha":"<hex>"}` (bearer-gated like the other mutators). It cannot do the work
itself — it runs `User=sando`, `NoNewPrivileges`, `ProtectSystem=strict`, so it
can neither write `/usr/local/bin/sandod` nor restart its own service. It only
*triggers* the root oneshot `sando-update@<sha>.service`, which the `sando` user
is allowed to `start` (and nothing else) by `10-sando-update.rules`. That unit
runs `sando-self-update.sh`: it builds `sando/daemon` **as the sando user** in a
dedicated checkout (`/srv/sando/self-update`, never the operator's dev tree),
then — as root — installs the binary and `systemctl restart sandod`.

Three hardening properties gate that install (postmortem 2026-07-09 recovery):

- **Build source = the local bare repo** `/srv/sando/mnw.git`, never a remote
  fetch by the build. Building from `git@ssh.makenot.work` gave the `sando` user
  no git creds and broke the moment git hosting was down (postmortem #7).
  `sandod` itself fetches the deploy branch into that bare repo at the top of the
  `/self-update` handler, so a sha pushed minutes ago is reachable. Until 0.2.9
  the repo advanced only when a *server* build fetched during `/rebuild`, which
  chained the controller's currency to the server's release cadence and made a
  controller fix unshippable while the server was red. `sandod` remains the only
  writer of that repo, so the provenance seal below is unchanged.
- **Clean build** (`cargo clean` first) so no stale incremental object survives
  across shas. A reused pre-`node_health` `Gate` enum object once produced a
  binary that crash-looped on the current config.
- **`--check-config` self-test before the swap.** The freshly built binary must
  load + parse the live daemon config + topology (`sandod --check-config`, run as
  the sando user against `/etc/sando/sando-daemon.toml`). A binary that can't
  understand the current config fails here and is never installed — `sandod`
  keeps running on the old one instead of crash-looping on the new.

Trigger `/self-update` only when no MNW build/deploy is in flight — `sandod`
rejects it in that window anyway (the restart would SIGKILL an in-flight deploy),
but treat controller updates as a separate, deliberate op.

Verify a self-update landed by polling `/state`: the new `sandod_version` field
reports the running controller's package version (distinct from the tier
versions, which are the deployed product).

Do **not** poll `systemctl is-active sando-update@<sha>` for that. The unit
reports `activating` for the whole build, so `until ! systemctl is-active ...`
exits immediately and reads a running build as a finished one. Poll
`ActiveState` explicitly, or poll `/state` for the `sandod_version` you expect.

One-time install (as root), then every future controller deploy is one API call:

```sh
sudo install -d /usr/local/lib/sando
sudo install -m 0755 sando-self-update.sh /usr/local/lib/sando/sando-self-update.sh
sudo install -m 0644 sando-update@.service /etc/systemd/system/
sudo install -m 0644 10-sando-update.rules /etc/polkit-1/rules.d/
sudo systemctl daemon-reload
```

Bootstrap caveat: the *first* `sandod` carrying `/self-update` still has to be
installed by hand (build + `install` + `systemctl restart sandod`) — there is no
endpoint to call until it is running. Self-update is for every deploy after that.
Trigger + watch:

```sh
# The sha must be on the canonical remote's main — that is what sandod fetches
# and what the updater's ancestry check seals against. Read it from the remote,
# not from the working copy, which may be ahead of what was pushed.
SHA=$(git -C ~/Code/MNW ls-remote mnw main | cut -f1)
curl -sS -X POST "$BASE/self-update" -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $SANDO_API_TOKEN" -d "{\"sha\":\"$SHA\"}"
journalctl -u "sando-update@$SHA" -f
```

## Editing a node's env file

`/etc/mnw/makenotwork.env` must stay **mode 0640, owned root:<service user>**.
`bootstrap-node.sh` creates it that way and the deploy depends on it.

The trap, which cost a failed prod deploy on 2026-08-01: systemd reads
`EnvironmentFile=` as root before dropping to `User=`, so the running service
does not care about the mode. Sando's pre-swap config check is the only thing
that reads the file **as the deploy user**, because its whole job is to load the
config the way the service will before swapping the symlink. So a file rewritten
0600 leaves production serving happily while the next deploy is already broken,
and nothing says so until someone ships.

Rewriting the file through a temp file is what does it:

    # WRONG: the temp file carries root's 077 umask, and `mv` carries it over
    { cat /etc/mnw/makenotwork.env; echo "NEW_VAR=x"; } > /tmp/env && \
      mv /tmp/env /etc/mnw/makenotwork.env

`mv` replaces the inode, so the new file keeps the temp's 0600 and loses any ACL
the old one carried. Append in place, or restore the mode explicitly afterwards:

    # RIGHT: in-place, mode preserved
    echo "NEW_VAR=x" >> /etc/mnw/makenotwork.env

    # or, if the file must be rewritten
    install -m 0640 -o root -g "$SERVICE_USER" /tmp/env /etc/mnw/makenotwork.env

The deploy now fails with a readability probe naming the user, the mode and the
owner rather than a bare `Permission denied` from inside a generated script, so
this is recoverable in one read — but it is still a failed deploy.

## Rollback contract

A Sando rollback (canary rollback of a node, or an operator swapping the
`current` symlink back to an older release dir) restores **the binary and the
release contents only**. The database does not roll back:

- MNW migrates **forward** on boot (`makenotwork.service` runs pending migrations
  when it starts).
- `sqlx::migrate::Migrator` has no down path, and Sando never invokes one.

So rolling, say, `0.10.15` back to `0.10.14` **after** `0.10.15` applied a
migration leaves the old binary running against a newer schema. That is a
one-way door for any release that carries a migration: forward is safe, back is
not. Sando has no restore-to-prod path either — `/backup/fetch` pulls the prod
dump solely as input to the `migration_dry_run` gate, not to restore a live node.

**Operating rule.** Promote a migration-bearing release with
`{"bears_migration": true}`. That forces a fresh `manual_confirm` on the
predecessor tier before the advance, even on a tier that configures no confirm,
so the one-way advance is a conscious step. `hotfix` does not suppress it (it
skips only `burn_in`). To actually undo a migration-bearing release you must
restore the database from a backup by hand first; a symlink rollback alone will
serve a mismatched schema.

**Does the backup actually restore?** The `migration_dry_run` gate answers this
on every build, once per `[[migration_check]]`: it resets that check's scratch
database, restores the latest dump of the database the check is for, and runs
the migrator against it. A failed restore fails the gate. Both prod databases
are covered — `makenotwork` and `multithreaded`, each from its own dump under
`/srv/sando/backups/`. A check that names a scratch database of its own gets it
created by the daemon on first run, so adding one owes no step here. What is
*not* automated is a full restore-to-serving drill (restore into a throwaway
target and confirm the app boots and serves
against it) — that is tracked as an infra task, not wired into the pipeline.

## Companion services (deploying mnw-cli in lockstep)

`mnw-cli` (the public git-SSH server that proxies to `/api/internal/*`) shares
the server's internal-API contract but used to deploy from its own
`mnw-cli/deploy/deploy.sh`. It drifted two months out of lockstep and broke git
hosting the moment the server tightened that contract (0.10.14, postmortem #4).
Sando now builds and ships it in the **same promote** as the server:

- **Build** — `[[companion]]` in the daemon config lists crates to compile from
  the same worktree/sha as the server. Each is built after the server and staged
  into the release bundle as `companions/<name>`. A companion that fails to build
  fails the whole pipeline — that is the lockstep guarantee.
- **Deploy** — `[[tier.node.companion]]` on a node says which companions it
  installs, to what `install_path`, and which unit to restart. After the server
  is swapped and back up (mnw-cli is `After=makenotwork.service`), the node
  installs the staged binary and restarts the unit. testnot has no mnw-cli, so
  only `prod-1` declares it.

Deploy runs `sudo /usr/local/lib/mnw/install-companion.sh <src> <dst> <service>`
over the node's executor. The wrapper keeps the deploy user's sudo grant to one
auditable script (it validates: src inside a release bundle, dst under `/opt`,
service a bare `*.service`) rather than a broad `install`/`systemctl` grant.

The grant names every deploy user Sando SSHes as — `makenotwork` for prod-1,
`deploy` for testnot-1 — so the file installs verbatim on any node and needs no
hand-edit at install time. sudoers accepts a user that is not present on the box,
so the other node's line is inert rather than an error. A node added later needs
its `ssh_target` user added to `mnw-companion.sudoers`.

One-time per node that hosts a companion (currently prod-1), as root:

```sh
sudo install -d /usr/local/lib/mnw
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion
sudo visudo -cf /etc/sudoers.d/mnw-companion   # validate before trusting it
```

After that, every promote that reaches the node ships the server and its
companions together; no separate `mnw-cli` deploy step. The legacy
`mnw-cli/deploy/deploy.sh` is retired once the first lockstep prod promote lands.

## testnot.work staging

testnot is gated app-side to Fan+/creator accounts
(`ACCESS_GATE=fan_plus_or_creator`). It exists so creators and Fan+ members can
preview upcoming features, and to back the pre-cutover migration dry-run.

**Source of truth: the seeded example catalog** (`mnw-testnot-seed.sh`). testnot
no longer mirrors production. Instead it runs a self-contained catalog of
fabricated `@example.test` creators and public-domain items, built by the app's
`--seed-examples` flow — so no prod-derived data lives on the staging box. The
script stops the app, resets the schema (recreating `public` owned by the app
role — PG15+ otherwise blocks the app role's boot migrations), then runs the
binary once with `--seed-examples`, which migrates the empty schema and runs the
guarded seed before exiting, and restarts the app. It is idempotent — re-run it
any time to reset testnot to the fixed catalog. There is no pause-flag; the
catalog is stable by construction.

Media (previews/downloads) attaches only when object storage is configured in
`/etc/mnw/makenotwork.env` (`S3_*`, plus `S3_PUBLIC_BUCKET` + `CDN_BASE_URL` for
covers). Until MinIO is stood up on testnot, items seed **hidden** (creators,
projects, blog posts, and follow counts still show).

Install on the Sando host:

```sh
sudo install -m 0755 mnw-testnot-seed.sh /usr/local/bin/mnw-testnot-seed.sh
sudo install -m 0755 mnw-testnot-smoke.sh /usr/local/bin/mnw-testnot-smoke.sh
sudo install -m 0644 mnw-testnot-seed.service /etc/systemd/system/
sudo systemctl daemon-reload
```

Install both. The seed script ends by `exec`ing the smoke check from its own
directory, so a seed installed without its sibling reseeds and then fails at the
last step.

Reset testnot any time: `sudo /usr/local/bin/mnw-testnot-seed.sh`
(or `sudo systemctl start mnw-testnot-seed.service`).

A reseed is not finished when the health check passes. Healthy is not the same
as worth showing: every page can return 200 while the catalog is empty, which is
how nine of eleven items stayed invisible for weeks. The smoke check is the part
that notices, and it runs automatically at the end of every seed. Run it alone
whenever you want the same answer: `./mnw-testnot-smoke.sh`. Set
`SKIP_CONTENT_SMOKE=1` on the seed to skip it while the catalog is deliberately
mid-change.

Its floors are sealed baselines in the same spirit as `tests/test_hygiene.rs`:
they freeze what is true today and fail on a new violation. Move one only in the
improving direction, and never loosen one to make a run pass.

The seed flow is the only way testnot gets its data. The prod-restore refresh it
replaced was deleted on 2026-07-22; testnot holds no prod-derived data and there
is no path that puts any there.

**Redeploy the binary** (no Sando integration yet — manual): build the release
on the Sando host, then over Tailscale SSH as root on the target, copy the
current release dir to a new one, stream the new `makenotwork` binary in, stream
a tar of `static/` + `docs/` (`docs/` = `server/site-docs/{public,examples}` +
`server/docs/business/assumptions.toml`, mirroring
`server/deploy/archive/deploy.sh.legacy`),
flip the `current` symlink, and `systemctl restart makenotwork.service` (which
boot-migrates). Old release dirs are kept for rollback.

Cert, the makenotwork Postgres password, the Caddyfile, and the operator
runbook for the root-level node setup are in `_private/infra/testnot/`.