| 1 |
# Deploying pom |
| 2 |
|
| 3 |
pom is **built by Bento and shipped by Sando**, and not by hand. Bento builds |
| 4 |
and packages; Sando decides whether a thing advances a stage, and performs the |
| 5 |
advance. Design: wiki `sando-bento-boundary`. |
| 6 |
|
| 7 |
Neither controller could do this alone. `build::run` compiles only on the one |
| 8 |
host `build_host` names, which is the never-build-on-prod invariant and also |
| 9 |
makes Sando single-architecture; pom runs on astra (aarch64) and on the Hetzner |
| 10 |
box (x86_64), so Sando could never build half of a release without breaking its |
| 11 |
own rule. Bento fans out native builds across both machines and cannot honestly |
| 12 |
gate an artifact in a place it does not know about. So `pom/bento.toml` declares |
| 13 |
`kind = "service"` with no `[[deploy]]` tables, and the daemon's `[handoff.pom]` |
| 14 |
carries the collected binary to sandod, which owns install, restart, health and |
| 15 |
rollback from there. |
| 16 |
|
| 17 |
`deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the |
| 18 |
native-per-architecture rule, with a cargo-zigbuild that is not installed on |
| 19 |
fw13), reached Hetzner on port 2200 which refuses this machine's key, and copied |
| 20 |
config and the unit file over the live ones with no diff and no backup. It did |
| 21 |
not work as written when 0.4.0 went out; that deploy was done by hand. |
| 22 |
|
| 23 |
## Running a deploy |
| 24 |
|
| 25 |
Bump the version in `Cargo.toml`, commit, tag `pom-v<version>`, push, then |
| 26 |
trigger the build through Bento for both targets. The `/deploy` skill has the |
| 27 |
procedure. |
| 28 |
|
| 29 |
Bento's half, per target (`dist/recipes/linux.rhai`): |
| 30 |
|
| 31 |
1. Pin the checkout to the release tag, and check both hosts report one commit. |
| 32 |
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host. |
| 33 |
3. `cargo build --release`. |
| 34 |
4. Assert `pom --version` matches the tag. |
| 35 |
5. `collect` the binary. The daemon hashes it, writes the artifact record beside |
| 36 |
it, stages it into `/srv/sando/pom/releases/staging` over ssh, and calls |
| 37 |
`POST /apps/pom/intake`. |
| 38 |
|
| 39 |
The recipe stops there. There is no `glibc_check` in it any more and no health |
| 40 |
poll: both are evidence about the artifact *in a place*, which a build host |
| 41 |
cannot honestly produce. They moved to Sando's side, where the node is known. |
| 42 |
|
| 43 |
Sando's half, one tier at a time (`sando/pom-topology.toml`): |
| 44 |
|
| 45 |
|
| 46 |
|
| 47 |
| `host` | none — the tier an intake lands on | none; Bento's verdicts ride in the artifact record | |
| 48 |
| `astra` | astra, aarch64 | `node_health`, `burn_in` 24h, `manual_confirm` | |
| 49 |
| `hetzner` | alpha-west-1, x86_64 | `node_health`, `manual_confirm` | |
| 50 |
|
| 51 |
``` |
| 52 |
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \ |
| 53 |
"$SANDO_DAEMON/apps/pom/promote/astra" |
| 54 |
``` |
| 55 |
|
| 56 |
Note the `/apps/pom` prefix. The unprefixed routes address the default product, |
| 57 |
which is `mnw`; `POST /promote/astra` from muscle memory promotes the wrong |
| 58 |
thing. Read state the same way: `GET /apps/pom/status.json`. |
| 59 |
|
| 60 |
astra goes first because it is the instance that matters least if pom is down for |
| 61 |
a moment: it watches, it does not serve anyone. pom watches its own deploy, so |
| 62 |
the restart takes one watcher down; the Hetzner instance keeps watching, and |
| 63 |
health is read from the peer over the mesh. Never both at once. |
| 64 |
|
| 65 |
## One-time node setup |
| 66 |
|
| 67 |
A node has to be prepared before its first promote, and the failure if it is not |
| 68 |
is silent rather than loud: sandod rsyncs the bundle, swaps `current`, restarts |
| 69 |
a unit still pointing at `/usr/local/bin/pom`, and `node_health` passes on the |
| 70 |
old binary. Sando reports the version shipped and nothing shipped. |
| 71 |
|
| 72 |
`bootstrap-pom-node.sh` is that preparation, idempotent, run on the node as root: |
| 73 |
|
| 74 |
``` |
| 75 |
sudo DEPLOY_USER=max \ |
| 76 |
SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \ |
| 77 |
./bootstrap-pom-node.sh |
| 78 |
``` |
| 79 |
|
| 80 |
`DEPLOY_USER` is the user in that node's `ssh_target`: `max` on astra, `root` on |
| 81 |
the Hetzner box. It creates `/opt/pom` owned by that user, grants it |
| 82 |
`systemctl reload-or-restart pom.service` through one scoped sudoers line, seeds |
| 83 |
the currently-installed binary as the first release so the unit never points at a |
| 84 |
dangling symlink, and moves `ExecStart` to `/opt/pom/current/pom` through a |
| 85 |
drop-in. The node comes out on the version it went in on, deployable. |
| 86 |
|
| 87 |
It also has to be reachable: sandod runs as the `sando` user on fw13, so that |
| 88 |
user's ssh (its key, and its `known_hosts`) is what has to reach each node, not |
| 89 |
max's. |
| 90 |
|
| 91 |
## What a deploy does not touch |
| 92 |
|
| 93 |
**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the |
| 94 |
live config on prod once carried a `[targets.mnw.tests]` block this repo did not |
| 95 |
have. `deploy.sh` would have silently deleted it. Config is a separate, |
| 96 |
deliberate act; nothing in the pipeline goes near it. |
| 97 |
|
| 98 |
**The unit file.** Same reasoning. A hardened unit that has drifted from the |
| 99 |
repo is a question for a human, not something a binary deploy overwrites. That |
| 100 |
is why the bootstrap moves `ExecStart` in a drop-in: one reversible file beside |
| 101 |
the unit, rather than an edit of it. |
| 102 |
|
| 103 |
## The database path is config, not environment |
| 104 |
|
| 105 |
Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and |
| 106 |
that is now the only thing deciding where the database is. It has to be stated, |
| 107 |
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does |
| 108 |
not: before it was configured, `pom serve` under systemd and `pom test` run by |
| 109 |
hand as the `pom` user opened two different files on the same host. The suites |
| 110 |
really ran and really passed, into a database nothing serves, while |
| 111 |
`/status.json` said no tests had ever run. |
| 112 |
|
| 113 |
The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary |
| 114 |
finds the same file. A current binary ignores it. |
| 115 |
|
| 116 |
A first install has to create the database once, since opening a missing one is |
| 117 |
now an error rather than a silent create: |
| 118 |
|
| 119 |
``` |
| 120 |
sudo -u pom pom --init --config /etc/pom/pom.toml status |
| 121 |
``` |
| 122 |
|
| 123 |
## The retired Bento install path |
| 124 |
|
| 125 |
`install-service.sh` and `bento-deploy.sudoers` are what Bento used when it |
| 126 |
installed pom itself: a single script-guarded sudo grant, staging under |
| 127 |
`/var/tmp/bento-deploy`, atomic `install(1)` to `/usr/local/bin/pom` with the |
| 128 |
previous binary kept as `.prev`. Nothing in the pipeline calls them any more. |
| 129 |
|
| 130 |
They are still here, and still installed on both nodes, because until a Sando |
| 131 |
promote has actually landed on a node, `/usr/local/bin/pom` is what that node is |
| 132 |
running and `.prev` is its rollback. Remove both files, and the grant and |
| 133 |
installer on each host, in a separate pass once each node has taken a release |
| 134 |
through Sando. |
| 135 |
|
| 136 |
## The test runner on astra |
| 137 |
|
| 138 |
astra is the only instance that runs test suites, and it runs them itself rather |
| 139 |
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can |
| 140 |
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop |
| 141 |
to reach (Tailscale SSH does not intercept a node connecting to itself, which is |
| 142 |
why the config's old `ssh = "max@<astra>"` failed `Connection refused` and no |
| 143 |
suite ever ran). `TestsConfig.ssh` is therefore optional; omitted, the command |
| 144 |
runs as a local child. |
| 145 |
|
| 146 |
Everything the runner needs lives under `/var/lib/pom`, the one path the hardened |
| 147 |
unit can write: |
| 148 |
|
| 149 |
|
| 150 |
|
| 151 |
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here | |
| 152 |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here | |
| 153 |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` | |
| 154 |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` | |
| 155 |
|
| 156 |
The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any |
| 157 |
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the |
| 158 |
clones can fetch from the mirrors. |
| 159 |
|
| 160 |
`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records |
| 161 |
that layout, and both Tauri apps carry a symlink that reaches across it |
| 162 |
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`). |
| 163 |
Flatten the clones and it dangles, and the app's `build.rs` panics on a read. |
| 164 |
|
| 165 |
Postgres notes, all of which cause failures that look like something else: |
| 166 |
`PGUSER=pom` is required because sqlx cannot resolve a username inside the |
| 167 |
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need |
| 168 |
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is |
| 169 |
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than |
| 170 |
compiling against the committed `.sqlx` cache, which goes stale silently whenever |
| 171 |
a migration lands without a `cargo sqlx prepare`. |
| 172 |
|
| 173 |
**The host timezone is load-bearing.** astra was on `America/Los_Angeles` and is |
| 174 |
now on `America/Denver`, matching fw13. goingson's |
| 175 |
`a_relative_event_keeps_its_wall_clock_across_a_move` asserts a civil time that |
| 176 |
only holds at UTC-6, and it reads `/etc/localtime` rather than `TZ`, so exporting |
| 177 |
`TZ` in this script does not move it — only the host setting does. That is a |
| 178 |
latent fragility in the test worth fixing at the source; until then a CI host on |
| 179 |
any other zone reports goingson red for no reason. |
| 180 |
|
| 181 |
To reproduce a red suite by hand: |
| 182 |
|
| 183 |
``` |
| 184 |
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw |
| 185 |
``` |
| 186 |
|
| 187 |
A failure that only appears under `systemd-run` with the unit's properties is a |
| 188 |
hardening problem, not a test problem. |
| 189 |
|
| 190 |
## Rollback |
| 191 |
|
| 192 |
Sando owns it, per tier: |
| 193 |
|
| 194 |
``` |
| 195 |
curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \ |
| 196 |
"$SANDO_DAEMON/apps/pom/rollback/astra" |
| 197 |
``` |
| 198 |
|
| 199 |
That is the whole gain from the move. Release dirs are named for their content |
| 200 |
digest rather than their version, so every release the node has kept is a |
| 201 |
rollback target and the horizon is as deep as the GC allows. What pom had before |
| 202 |
was a single `<install-path>.prev` file: one version back, overwritten by the |
| 203 |
next deploy. |
| 204 |
|
| 205 |
A node that has not taken a Sando release yet is still on the old path, and |
| 206 |
rolling it back is putting `.prev` back by hand: |
| 207 |
|
| 208 |
``` |
| 209 |
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom |
| 210 |
sudo systemctl restart pom.service |
| 211 |
``` |
| 212 |
|