| 1 |
# Deploying pom |
| 2 |
|
| 3 |
pom is **built by Bento and shipped by Sando**, and not by hand. Bento builds |
| 4 |
and packages; Sando decides whether a thing advances a stage, and performs the |
| 5 |
advance. Design: wiki `sando-bento-boundary`. |
| 6 |
|
| 7 |
Neither controller could do this alone. `build::run` compiles only on the one |
| 8 |
host `build_host` names, which is the never-build-on-prod invariant and also |
| 9 |
makes Sando single-architecture; pom runs on astra (aarch64) and on the Hetzner |
| 10 |
box (x86_64), so Sando could never build half of a release without breaking its |
| 11 |
own rule. Bento fans out native builds across both machines and cannot honestly |
| 12 |
gate an artifact in a place it does not know about. So `pom/bento.toml` declares |
| 13 |
`kind = "service"` with no `[[deploy]]` tables, and the daemon's `[handoff.pom]` |
| 14 |
carries the collected binary to sandod, which owns install, restart, health and |
| 15 |
rollback from there. |
| 16 |
|
| 17 |
## Running a deploy |
| 18 |
|
| 19 |
Bump the version in `Cargo.toml`, commit, tag `pom-v<version>`, push, then |
| 20 |
trigger the build through Bento for both targets. The `/deploy` skill has the |
| 21 |
procedure. |
| 22 |
|
| 23 |
Bento's half, per target (`dist/recipes/linux.rhai`): |
| 24 |
|
| 25 |
1. Pin the checkout to the release tag, and check both hosts report one commit. |
| 26 |
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host. |
| 27 |
3. `cargo build --release`. |
| 28 |
4. Assert `pom --version` matches the tag. |
| 29 |
5. `collect` the binary. The daemon hashes it, writes the artifact record beside |
| 30 |
it, stages it into `/srv/sando/pom/releases/staging` over ssh, and calls |
| 31 |
`POST /apps/pom/intake`. |
| 32 |
|
| 33 |
The recipe stops there: no `glibc_check` and no health poll. Both are evidence |
| 34 |
about the artifact *in a place*, which a build host cannot honestly produce, so |
| 35 |
they live on Sando's side, where the node is known. |
| 36 |
|
| 37 |
Sando's half, one tier at a time (`sando/pom-topology.toml`): |
| 38 |
|
| 39 |
|
| 40 |
|
| 41 |
| `host` | none; the tier an intake lands on | none; Bento's verdicts ride in the artifact record | |
| 42 |
| `astra` | astra, aarch64 | `node_health`, `burn_in` 24h, `manual_confirm` | |
| 43 |
| `hetzner` | alpha-west-1, x86_64 | `node_health`, `manual_confirm` | |
| 44 |
|
| 45 |
``` |
| 46 |
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \ |
| 47 |
"$SANDO_DAEMON/apps/pom/promote/astra" |
| 48 |
``` |
| 49 |
|
| 50 |
Note the `/apps/pom` prefix. The unprefixed routes address the default product, |
| 51 |
which is `mnw`; `POST /promote/astra` from muscle memory promotes the wrong |
| 52 |
thing. Read state the same way: `GET /apps/pom/status.json`. |
| 53 |
|
| 54 |
astra goes first because it is the instance that matters least if pom is down for |
| 55 |
a moment: it watches, it does not serve anyone. pom watches its own deploy, so |
| 56 |
the restart takes one watcher down; the Hetzner instance keeps watching, and |
| 57 |
health is read from the peer over the mesh. Never both at once. |
| 58 |
|
| 59 |
## One-time node setup |
| 60 |
|
| 61 |
A node has to be prepared before its first promote, and the failure if it is not |
| 62 |
is silent rather than loud: sandod rsyncs the bundle, swaps `current`, restarts |
| 63 |
a unit still pointing at `/usr/local/bin/pom`, and `node_health` passes on the |
| 64 |
old binary. Sando reports the version shipped and nothing shipped. |
| 65 |
|
| 66 |
`bootstrap-pom-node.sh` is that preparation, idempotent, run on the node as root: |
| 67 |
|
| 68 |
``` |
| 69 |
sudo DEPLOY_USER=max \ |
| 70 |
SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \ |
| 71 |
./bootstrap-pom-node.sh |
| 72 |
``` |
| 73 |
|
| 74 |
`DEPLOY_USER` is the user in that node's `ssh_target`: `max` on astra, `root` on |
| 75 |
the Hetzner box. It creates `/opt/pom` owned by that user, grants it |
| 76 |
`systemctl reload-or-restart pom.service` through one scoped sudoers line, seeds |
| 77 |
the currently-installed binary as the first release so the unit never points at a |
| 78 |
dangling symlink, and moves `ExecStart` to `/opt/pom/current/pom` through a |
| 79 |
drop-in. The node comes out on the version it went in on, deployable. |
| 80 |
|
| 81 |
It also has to be reachable: sandod runs as the `sando` user on fw13, so that |
| 82 |
user's ssh (its key, and its `known_hosts`) is what has to reach each node, not |
| 83 |
max's. |
| 84 |
|
| 85 |
## What a deploy does not touch |
| 86 |
|
| 87 |
**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the |
| 88 |
live config on a node can carry blocks this repo does not have. Config is a |
| 89 |
separate, deliberate act; nothing in the pipeline goes near it. |
| 90 |
|
| 91 |
**The unit file.** Same reasoning. A hardened unit that has drifted from the |
| 92 |
repo is a question for a human, not something a binary deploy overwrites. That |
| 93 |
is why the bootstrap moves `ExecStart` in a drop-in: one reversible file beside |
| 94 |
the unit, rather than an edit of it. |
| 95 |
|
| 96 |
## The database path is config, not environment |
| 97 |
|
| 98 |
Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and |
| 99 |
that is the only thing deciding where the database is. It has to be stated, |
| 100 |
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does |
| 101 |
not, so an unconfigured `pom serve` under systemd and a hand-run `pom test` as |
| 102 |
the `pom` user open two different files on the same host, and a suite passes |
| 103 |
into a database nothing serves. |
| 104 |
|
| 105 |
The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary |
| 106 |
finds the same file. A current binary ignores it. |
| 107 |
|
| 108 |
A first install has to create the database once, since opening a missing one is |
| 109 |
an error rather than a silent create: |
| 110 |
|
| 111 |
``` |
| 112 |
sudo -u pom pom --init --config /etc/pom/pom.toml status |
| 113 |
``` |
| 114 |
|
| 115 |
## The legacy Bento install path |
| 116 |
|
| 117 |
`install-service.sh` and `bento-deploy.sudoers` install pom directly: a single |
| 118 |
script-guarded sudo grant, staging under `/var/tmp/bento-deploy`, atomic |
| 119 |
`install(1)` to `/usr/local/bin/pom` with the previous binary kept as `.prev`. |
| 120 |
Nothing in the pipeline calls them. |
| 121 |
|
| 122 |
They stay here, and installed on both nodes, because until a Sando promote has |
| 123 |
landed on a node, `/usr/local/bin/pom` is what that node runs and `.prev` is its |
| 124 |
rollback. Remove both files, and the grant and |
| 125 |
installer on each host, in a separate pass once each node has taken a release |
| 126 |
through Sando. |
| 127 |
|
| 128 |
## The test runner on astra |
| 129 |
|
| 130 |
astra is the only instance that runs test suites, and it runs them itself rather |
| 131 |
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can |
| 132 |
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop |
| 133 |
to reach (Tailscale SSH does not intercept a node connecting to itself, so an |
| 134 |
`ssh = "max@<astra>"` entry fails `Connection refused`). `TestsConfig.ssh` is |
| 135 |
therefore optional; omitted, the command runs as a local child. |
| 136 |
|
| 137 |
Everything the runner needs lives under `/var/lib/pom`, the one path the hardened |
| 138 |
unit can write: |
| 139 |
|
| 140 |
|
| 141 |
|
| 142 |
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here | |
| 143 |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here | |
| 144 |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` | |
| 145 |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` | |
| 146 |
|
| 147 |
The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any |
| 148 |
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the |
| 149 |
clones can fetch from the mirrors. |
| 150 |
|
| 151 |
`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records |
| 152 |
that layout, and both Tauri apps carry a symlink that reaches across it |
| 153 |
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`). |
| 154 |
Flatten the clones and it dangles, and the app's `build.rs` panics on a read. |
| 155 |
|
| 156 |
Postgres notes, all of which cause failures that look like something else: |
| 157 |
`PGUSER=pom` is required because sqlx cannot resolve a username inside the |
| 158 |
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need |
| 159 |
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is |
| 160 |
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than |
| 161 |
compiling against the committed `.sqlx` cache, which goes stale silently whenever |
| 162 |
a migration lands without a `cargo sqlx prepare`. |
| 163 |
|
| 164 |
**The host timezone is load-bearing.** astra is on `America/Denver`, matching |
| 165 |
fw13. goingson's `a_relative_event_keeps_its_wall_clock_across_a_move` asserts a |
| 166 |
civil time that only holds at UTC-6, and it reads `/etc/localtime` rather than |
| 167 |
`TZ`, so exporting `TZ` in this script does not move it; only the host setting |
| 168 |
does. A CI host on any other zone reports goingson red for no reason. |
| 169 |
|
| 170 |
To reproduce a red suite by hand: |
| 171 |
|
| 172 |
``` |
| 173 |
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw |
| 174 |
``` |
| 175 |
|
| 176 |
A failure that only appears under `systemd-run` with the unit's properties is a |
| 177 |
hardening problem, not a test problem. |
| 178 |
|
| 179 |
## Rollback |
| 180 |
|
| 181 |
Sando owns it, per tier: |
| 182 |
|
| 183 |
``` |
| 184 |
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \ |
| 185 |
"$SANDO_DAEMON/apps/pom/rollback/astra" |
| 186 |
``` |
| 187 |
|
| 188 |
Release dirs are named for their content digest rather than their version, so |
| 189 |
every release the node has kept is a rollback target and the horizon is as deep |
| 190 |
as the GC allows. |
| 191 |
|
| 192 |
A node that has not taken a Sando release yet is still on the old path, and |
| 193 |
rolling it back is putting `.prev` back by hand: |
| 194 |
|
| 195 |
``` |
| 196 |
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom |
| 197 |
sudo systemctl restart pom.service |
| 198 |
``` |
| 199 |
|