# Deploying pom

pom ships through **Bento**, not Sando, and not by hand.

Sando cannot do this job. `sando-daemon.toml` sets `build_host = "fw13"` and
`build::run` refuses to compile anywhere else, which is the never-build-on-prod
invariant and also makes Sando single-architecture. pom runs on astra (aarch64)
and on the Hetzner box (x86_64), so Sando could never build half of this release
without breaking its own rule. Bento already fans out native builds across both
hosts, so pom is a Bento recipe.

`deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the
native-per-architecture rule, with a cargo-zigbuild that is not installed on
fw13), reached Hetzner on port 2200 which refuses this machine's key, and copied
config and the unit file over the live ones with no diff and no backup. It did
not work as written when 0.4.0 went out; that deploy was done by hand.

## Running a deploy

Bump the version in `Cargo.toml`, commit, tag `v<version>`, push, then trigger
the build through Bento for both targets. The `/deploy` skill has the procedure.

What the recipe does, per target (`dist/recipes/linux.rhai`):

1. Pin the checkout to the release tag, and check both hosts report one commit.
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host.
3. `cargo build --release`.
4. Compare the binary's highest `GLIBC_` symbol against the service host's
   `ldd --version`, and assert `pom --version` matches the tag.
5. Stage the binary on the service host and call the privileged installer.
6. Poll `/api/health` until the restarted instance answers, then assert the
   running binary reports the version that was just installed.

The two instances go one at a time, and `require_all_targets` keeps the release
from counting as done until both are green. pom watches its own deploy, so the
restart takes one watcher down for a moment; the other stays up and keeps
watching. Never both at once.

## What a deploy does not touch

**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the
live config on prod once carried a `[targets.mnw.tests]` block this repo did not
have. `deploy.sh` would have silently deleted it. Config is a separate,
deliberate act; the installer refuses to go near it.

**The unit file.** Same reasoning. A hardened unit that has drifted from the
repo is a question for a human, not something a binary deploy overwrites.

## The database path is config, not environment

Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and
that is now the only thing deciding where the database is. It has to be stated,
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does
not: before it was configured, `pom serve` under systemd and `pom test` run by
hand as the `pom` user opened two different files on the same host. The suites
really ran and really passed, into a database nothing serves, while
`/status.json` said no tests had ever run.

The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary
finds the same file. A current binary ignores it.

A first install has to create the database once, since opening a missing one is
now an error rather than a silent create:

```
sudo -u pom pom --init --config /etc/pom/pom.toml status
```

## One-time host setup

Each host that runs pom needs the installer and its scoped sudo grant:

```
sudo install -d /usr/local/lib/bento
sudo install -m 0755 install-service.sh /usr/local/lib/bento/install-service.sh
sudo install -m 0440 bento-deploy.sudoers /etc/sudoers.d/bento-deploy
sudo visudo -cf /etc/sudoers.d/bento-deploy
```

Edit the sudoers file first so the user matches how Bento reaches that host:
`max` on astra, `root` over Tailscale SSH on the Hetzner box. The grant covers
one script and nothing else; the script bounds its own arguments (source under
`/var/tmp/bento-deploy`, destination under `/usr/local/bin`, a bare `*.service`
unit), so it is a script-guarded grant rather than a broad
`install` + `systemctl` one.

## The test runner on astra

astra is the only instance that runs test suites, and it runs them itself rather
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop
to reach (Tailscale SSH does not intercept a node connecting to itself, which is
why the config's old `ssh = "max@<astra>"` failed `Connection refused` and no
suite ever ran). `TestsConfig.ssh` is therefore optional; omitted, the command
runs as a local child.

Everything the runner needs lives under `/var/lib/pom`, the one path the hardened
unit can write:

| What | Where | Source |
|------|-------|--------|
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` |

The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the
clones can fetch from the mirrors.

`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records
that layout, and both Tauri apps carry a symlink that reaches across it
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`).
Flatten the clones and it dangles, and the app's `build.rs` panics on a read.

Postgres notes, all of which cause failures that look like something else:
`PGUSER=pom` is required because sqlx cannot resolve a username inside the
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than
compiling against the committed `.sqlx` cache, which goes stale silently whenever
a migration lands without a `cargo sqlx prepare`.

**The host timezone is load-bearing.** astra was on `America/Los_Angeles` and is
now on `America/Denver`, matching fw13. goingson's
`a_relative_event_keeps_its_wall_clock_across_a_move` asserts a civil time that
only holds at UTC-6, and it reads `/etc/localtime` rather than `TZ`, so exporting
`TZ` in this script does not move it — only the host setting does. That is a
latent fragility in the test worth fixing at the source; until then a CI host on
any other zone reports goingson red for no reason.

To reproduce a red suite by hand:

```
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw
```

A failure that only appears under `systemd-run` with the unit's properties is a
hardening problem, not a test problem.

## Rollback

The installer keeps the previous binary as `<install-path>.prev`. Rolling back
is putting it back and restarting:

```
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom
sudo systemctl restart pom.service
```