# Deploying pom

pom is **built by Bento and shipped by Sando**, and not by hand. Bento builds
and packages; Sando decides whether a thing advances a stage, and performs the
advance. Design: wiki `sando-bento-boundary`.

Neither controller could do this alone. `build::run` compiles only on the one
host `build_host` names, which is the never-build-on-prod invariant and also
makes Sando single-architecture; pom runs on astra (aarch64) and on the Hetzner
box (x86_64), so Sando could never build half of a release without breaking its
own rule. Bento fans out native builds across both machines and cannot honestly
gate an artifact in a place it does not know about. So `pom/bento.toml` declares
`kind = "service"` with no `[[deploy]]` tables, and the daemon's `[handoff.pom]`
carries the collected binary to sandod, which owns install, restart, health and
rollback from there.

`deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the
native-per-architecture rule, with a cargo-zigbuild that is not installed on
fw13), reached Hetzner on port 2200 which refuses this machine's key, and copied
config and the unit file over the live ones with no diff and no backup. It did
not work as written when 0.4.0 went out; that deploy was done by hand.

## Running a deploy

Bump the version in `Cargo.toml`, commit, tag `pom-v<version>`, push, then
trigger the build through Bento for both targets. The `/deploy` skill has the
procedure.

Bento's half, per target (`dist/recipes/linux.rhai`):

1. Pin the checkout to the release tag, and check both hosts report one commit.
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host.
3. `cargo build --release`.
4. Assert `pom --version` matches the tag.
5. `collect` the binary. The daemon hashes it, writes the artifact record beside
   it, stages it into `/srv/sando/pom/releases/staging` over ssh, and calls
   `POST /apps/pom/intake`.

The recipe stops there. There is no `glibc_check` in it any more and no health
poll: both are evidence about the artifact *in a place*, which a build host
cannot honestly produce. They moved to Sando's side, where the node is known.

Sando's half, one tier at a time (`sando/pom-topology.toml`):

| Tier | Node | Gates |
|------|------|-------|
| `host` | none — the tier an intake lands on | none; Bento's verdicts ride in the artifact record |
| `astra` | astra, aarch64 | `node_health`, `burn_in` 24h, `manual_confirm` |
| `hetzner` | alpha-west-1, x86_64 | `node_health`, `manual_confirm` |

```
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
    "$SANDO_DAEMON/apps/pom/promote/astra"
```

Note the `/apps/pom` prefix. The unprefixed routes address the default product,
which is `mnw`; `POST /promote/astra` from muscle memory promotes the wrong
thing. Read state the same way: `GET /apps/pom/status.json`.

astra goes first because it is the instance that matters least if pom is down for
a moment: it watches, it does not serve anyone. pom watches its own deploy, so
the restart takes one watcher down; the Hetzner instance keeps watching, and
health is read from the peer over the mesh. Never both at once.

## One-time node setup

A node has to be prepared before its first promote, and the failure if it is not
is silent rather than loud: sandod rsyncs the bundle, swaps `current`, restarts
a unit still pointing at `/usr/local/bin/pom`, and `node_health` passes on the
old binary. Sando reports the version shipped and nothing shipped.

`bootstrap-pom-node.sh` is that preparation, idempotent, run on the node as root:

```
sudo DEPLOY_USER=max \
    SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \
    ./bootstrap-pom-node.sh
```

`DEPLOY_USER` is the user in that node's `ssh_target`: `max` on astra, `root` on
the Hetzner box. It creates `/opt/pom` owned by that user, grants it
`systemctl reload-or-restart pom.service` through one scoped sudoers line, seeds
the currently-installed binary as the first release so the unit never points at a
dangling symlink, and moves `ExecStart` to `/opt/pom/current/pom` through a
drop-in. The node comes out on the version it went in on, deployable.

It also has to be reachable: sandod runs as the `sando` user on fw13, so that
user's ssh (its key, and its `known_hosts`) is what has to reach each node, not
max's.

## What a deploy does not touch

**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the
live config on prod once carried a `[targets.mnw.tests]` block this repo did not
have. `deploy.sh` would have silently deleted it. Config is a separate,
deliberate act; nothing in the pipeline goes near it.

**The unit file.** Same reasoning. A hardened unit that has drifted from the
repo is a question for a human, not something a binary deploy overwrites. That
is why the bootstrap moves `ExecStart` in a drop-in: one reversible file beside
the unit, rather than an edit of it.

## The database path is config, not environment

Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and
that is now the only thing deciding where the database is. It has to be stated,
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does
not: before it was configured, `pom serve` under systemd and `pom test` run by
hand as the `pom` user opened two different files on the same host. The suites
really ran and really passed, into a database nothing serves, while
`/status.json` said no tests had ever run.

The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary
finds the same file. A current binary ignores it.

A first install has to create the database once, since opening a missing one is
now an error rather than a silent create:

```
sudo -u pom pom --init --config /etc/pom/pom.toml status
```

## The retired Bento install path

`install-service.sh` and `bento-deploy.sudoers` are what Bento used when it
installed pom itself: a single script-guarded sudo grant, staging under
`/var/tmp/bento-deploy`, atomic `install(1)` to `/usr/local/bin/pom` with the
previous binary kept as `.prev`. Nothing in the pipeline calls them any more.

They are still here, and still installed on both nodes, because until a Sando
promote has actually landed on a node, `/usr/local/bin/pom` is what that node is
running and `.prev` is its rollback. Remove both files, and the grant and
installer on each host, in a separate pass once each node has taken a release
through Sando.

## The test runner on astra

astra is the only instance that runs test suites, and it runs them itself rather
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop
to reach (Tailscale SSH does not intercept a node connecting to itself, which is
why the config's old `ssh = "max@<astra>"` failed `Connection refused` and no
suite ever ran). `TestsConfig.ssh` is therefore optional; omitted, the command
runs as a local child.

Everything the runner needs lives under `/var/lib/pom`, the one path the hardened
unit can write:

| What | Where | Source |
|------|-------|--------|
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` |

The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the
clones can fetch from the mirrors.

`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records
that layout, and both Tauri apps carry a symlink that reaches across it
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`).
Flatten the clones and it dangles, and the app's `build.rs` panics on a read.

Postgres notes, all of which cause failures that look like something else:
`PGUSER=pom` is required because sqlx cannot resolve a username inside the
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than
compiling against the committed `.sqlx` cache, which goes stale silently whenever
a migration lands without a `cargo sqlx prepare`.

**The host timezone is load-bearing.** astra was on `America/Los_Angeles` and is
now on `America/Denver`, matching fw13. goingson's
`a_relative_event_keeps_its_wall_clock_across_a_move` asserts a civil time that
only holds at UTC-6, and it reads `/etc/localtime` rather than `TZ`, so exporting
`TZ` in this script does not move it — only the host setting does. That is a
latent fragility in the test worth fixing at the source; until then a CI host on
any other zone reports goingson red for no reason.

To reproduce a red suite by hand:

```
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw
```

A failure that only appears under `systemd-run` with the unit's properties is a
hardening problem, not a test problem.

## Rollback

Sando owns it, per tier:

```
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
    "$SANDO_DAEMON/apps/pom/rollback/astra"
```

That is the whole gain from the move. Release dirs are named for their content
digest rather than their version, so every release the node has kept is a
rollback target and the horizon is as deep as the GC allows. What pom had before
was a single `<install-path>.prev` file: one version back, overwritten by the
next deploy.

A node that has not taken a Sando release yet is still on the old path, and
rolling it back is putting `.prev` back by hand:

```
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom
sudo systemctl restart pom.service
```