# Deploying pom

pom is **built by Bento and shipped by Sando**, and not by hand. Bento builds
and packages; Sando decides whether a thing advances a stage, and performs the
advance. Design: wiki `sando-bento-boundary`.

Neither controller could do this alone. `build::run` compiles only on the one
host `build_host` names, which is the never-build-on-prod invariant and also
makes Sando single-architecture; pom runs on astra (aarch64) and on the Hetzner
box (x86_64), so Sando could never build half of a release without breaking its
own rule. Bento fans out native builds across both machines and cannot honestly
gate an artifact in a place it does not know about. So `pom/bento.toml` declares
`kind = "service"` with no `[[deploy]]` tables, and the daemon's `[handoff.pom]`
carries the collected binary to sandod, which owns install, restart, health and
rollback from there.

## Running a deploy

Bump the version in `Cargo.toml`, commit, tag `pom-v<version>`, push, then
trigger the build through Bento for both targets. The `/deploy` skill has the
procedure.

Bento's half, per target (`dist/recipes/linux.rhai`):

1. Pin the checkout to the release tag, and check both hosts report one commit.
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host.
3. `cargo build --release`.
4. Assert `pom --version` matches the tag.
5. `collect` the binary. The daemon hashes it, writes the artifact record beside
   it, stages it into `/srv/sando/pom/releases/staging` over ssh, and calls
   `POST /apps/pom/intake`.

The recipe stops there: no `glibc_check` and no health poll. Both are evidence
about the artifact *in a place*, which a build host cannot honestly produce, so
they live on Sando's side, where the node is known.

Sando's half, one tier at a time (`sando/pom-topology.toml`):

| Tier | Node | Gates |
|------|------|-------|
| `host` | none; the tier an intake lands on | none; Bento's verdicts ride in the artifact record |
| `astra` | astra, aarch64 | `node_health`, `burn_in` 24h, `manual_confirm` |
| `hetzner` | alpha-west-1, x86_64 | `node_health`, `manual_confirm` |

```
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
    "$SANDO_DAEMON/apps/pom/promote/astra"
```

Note the `/apps/pom` prefix. The unprefixed routes address the default product,
which is `mnw`; `POST /promote/astra` from muscle memory promotes the wrong
thing. Read state the same way: `GET /apps/pom/status.json`.

astra goes first because it is the instance that matters least if pom is down for
a moment: it watches, it does not serve anyone. pom watches its own deploy, so
the restart takes one watcher down; the Hetzner instance keeps watching, and
health is read from the peer over the mesh. Never both at once.

## One-time node setup

A node has to be prepared before its first promote, and the failure if it is not
is silent rather than loud: sandod rsyncs the bundle, swaps `current`, restarts
a unit still pointing at `/usr/local/bin/pom`, and `node_health` passes on the
old binary. Sando reports the version shipped and nothing shipped.

`bootstrap-pom-node.sh` is that preparation, idempotent, run on the node as root:

```
sudo DEPLOY_USER=max \
    SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \
    ./bootstrap-pom-node.sh
```

`DEPLOY_USER` is the user in that node's `ssh_target`: `max` on astra, `root` on
the Hetzner box. It creates `/opt/pom` owned by that user, grants it
`systemctl reload-or-restart pom.service` through one scoped sudoers line, seeds
the currently-installed binary as the first release so the unit never points at a
dangling symlink, and moves `ExecStart` to `/opt/pom/current/pom` through a
drop-in. The node comes out on the version it went in on, deployable.

It also has to be reachable: sandod runs as the `sando` user on fw13, so that
user's ssh (its key, and its `known_hosts`) is what has to reach each node, not
max's.

## What a deploy does not touch

**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the
live config on a node can carry blocks this repo does not have. Config is a
separate, deliberate act; nothing in the pipeline goes near it.

**The unit file.** Same reasoning. A hardened unit that has drifted from the
repo is a question for a human, not something a binary deploy overwrites. That
is why the bootstrap moves `ExecStart` in a drop-in: one reversible file beside
the unit, rather than an edit of it.

## The database path is config, not environment

Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and
that is the only thing deciding where the database is. It has to be stated,
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does
not, so an unconfigured `pom serve` under systemd and a hand-run `pom test` as
the `pom` user open two different files on the same host, and a suite passes
into a database nothing serves.

The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary
finds the same file. A current binary ignores it.

A first install has to create the database once, since opening a missing one is
an error rather than a silent create:

```
sudo -u pom pom --init --config /etc/pom/pom.toml status
```

## The legacy Bento install path

`install-service.sh` and `bento-deploy.sudoers` install pom directly: a single
script-guarded sudo grant, staging under `/var/tmp/bento-deploy`, atomic
`install(1)` to `/usr/local/bin/pom` with the previous binary kept as `.prev`.
Nothing in the pipeline calls them.

They stay here, and installed on both nodes, because until a Sando promote has
landed on a node, `/usr/local/bin/pom` is what that node runs and `.prev` is its
rollback. Remove both files, and the grant and
installer on each host, in a separate pass once each node has taken a release
through Sando.

## The test runner on astra

astra is the only instance that runs test suites, and it runs them itself rather
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop
to reach (Tailscale SSH does not intercept a node connecting to itself, so an
`ssh = "max@<astra>"` entry fails `Connection refused`). `TestsConfig.ssh` is
therefore optional; omitted, the command runs as a local child.

Everything the runner needs lives under `/var/lib/pom`, the one path the hardened
unit can write:

| What | Where | Source |
|------|-------|--------|
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` |

The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the
clones can fetch from the mirrors.

`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records
that layout, and both Tauri apps carry a symlink that reaches across it
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`).
Flatten the clones and it dangles, and the app's `build.rs` panics on a read.

Postgres notes, all of which cause failures that look like something else:
`PGUSER=pom` is required because sqlx cannot resolve a username inside the
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than
compiling against the committed `.sqlx` cache, which goes stale silently whenever
a migration lands without a `cargo sqlx prepare`.

**The host timezone is load-bearing.** astra is on `America/Denver`, matching
fw13. goingson's `a_relative_event_keeps_its_wall_clock_across_a_move` asserts a
civil time that only holds at UTC-6, and it reads `/etc/localtime` rather than
`TZ`, so exporting `TZ` in this script does not move it; only the host setting
does. A CI host on any other zone reports goingson red for no reason.

To reproduce a red suite by hand:

```
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw
```

A failure that only appears under `systemd-run` with the unit's properties is a
hardening problem, not a test problem.

## Rollback

Sando owns it, per tier:

```
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
    "$SANDO_DAEMON/apps/pom/rollback/astra"
```

Release dirs are named for their content digest rather than their version, so
every release the node has kept is a rollback target and the horizon is as deep
as the GC allows.

A node that has not taken a Sando release yet is still on the old path, and
rolling it back is putting `.prev` back by hand:

```
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom
sudo systemctl restart pom.service
```