| 1 |
# Deploying pom |
| 2 |
|
| 3 |
pom ships through **Bento**, not Sando, and not by hand. |
| 4 |
|
| 5 |
Sando cannot do this job. `sando-daemon.toml` sets `build_host = "fw13"` and |
| 6 |
`build::run` refuses to compile anywhere else, which is the never-build-on-prod |
| 7 |
invariant and also makes Sando single-architecture. pom runs on astra (aarch64) |
| 8 |
and on the Hetzner box (x86_64), so Sando could never build half of this release |
| 9 |
without breaking its own rule. Bento already fans out native builds across both |
| 10 |
hosts, so pom is a Bento recipe. |
| 11 |
|
| 12 |
`deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the |
| 13 |
native-per-architecture rule, with a cargo-zigbuild that is not installed on |
| 14 |
fw13), reached Hetzner on port 2200 which refuses this machine's key, and copied |
| 15 |
config and the unit file over the live ones with no diff and no backup. It did |
| 16 |
not work as written when 0.4.0 went out; that deploy was done by hand. |
| 17 |
|
| 18 |
## Running a deploy |
| 19 |
|
| 20 |
Bump the version in `Cargo.toml`, commit, tag `v<version>`, push, then trigger |
| 21 |
the build through Bento for both targets. The `/deploy` skill has the procedure. |
| 22 |
|
| 23 |
What the recipe does, per target (`dist/recipes/linux.rhai`): |
| 24 |
|
| 25 |
1. Pin the checkout to the release tag, and check both hosts report one commit. |
| 26 |
2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host. |
| 27 |
3. `cargo build --release`. |
| 28 |
4. Compare the binary's highest `GLIBC_` symbol against the service host's |
| 29 |
`ldd --version`, and assert `pom --version` matches the tag. |
| 30 |
5. Stage the binary on the service host and call the privileged installer. |
| 31 |
6. Poll `/api/health` until the restarted instance answers, then assert the |
| 32 |
running binary reports the version that was just installed. |
| 33 |
|
| 34 |
The two instances go one at a time, and `require_all_targets` keeps the release |
| 35 |
from counting as done until both are green. pom watches its own deploy, so the |
| 36 |
restart takes one watcher down for a moment; the other stays up and keeps |
| 37 |
watching. Never both at once. |
| 38 |
|
| 39 |
## What a deploy does not touch |
| 40 |
|
| 41 |
**Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the |
| 42 |
live config on prod once carried a `[targets.mnw.tests]` block this repo did not |
| 43 |
have. `deploy.sh` would have silently deleted it. Config is a separate, |
| 44 |
deliberate act; the installer refuses to go near it. |
| 45 |
|
| 46 |
**The unit file.** Same reasoning. A hardened unit that has drifted from the |
| 47 |
repo is a question for a human, not something a binary deploy overwrites. |
| 48 |
|
| 49 |
## The database path is config, not environment |
| 50 |
|
| 51 |
Both instance configs here set `storage.db_path = "/var/lib/pom/pom.db"`, and |
| 52 |
that is now the only thing deciding where the database is. It has to be stated, |
| 53 |
because the unit sets `XDG_DATA_HOME=/var/lib` and an interactive login does |
| 54 |
not: before it was configured, `pom serve` under systemd and `pom test` run by |
| 55 |
hand as the `pom` user opened two different files on the same host. The suites |
| 56 |
really ran and really passed, into a database nothing serves, while |
| 57 |
`/status.json` said no tests had ever run. |
| 58 |
|
| 59 |
The unit still carries the `XDG_DATA_HOME` line so a rolled-back older binary |
| 60 |
finds the same file. A current binary ignores it. |
| 61 |
|
| 62 |
A first install has to create the database once, since opening a missing one is |
| 63 |
now an error rather than a silent create: |
| 64 |
|
| 65 |
``` |
| 66 |
sudo -u pom pom --init --config /etc/pom/pom.toml status |
| 67 |
``` |
| 68 |
|
| 69 |
## One-time host setup |
| 70 |
|
| 71 |
Each host that runs pom needs the installer and its scoped sudo grant: |
| 72 |
|
| 73 |
``` |
| 74 |
sudo install -d /usr/local/lib/bento |
| 75 |
sudo install -m 0755 install-service.sh /usr/local/lib/bento/install-service.sh |
| 76 |
sudo install -m 0440 bento-deploy.sudoers /etc/sudoers.d/bento-deploy |
| 77 |
sudo visudo -cf /etc/sudoers.d/bento-deploy |
| 78 |
``` |
| 79 |
|
| 80 |
Edit the sudoers file first so the user matches how Bento reaches that host: |
| 81 |
`max` on astra, `root` over Tailscale SSH on the Hetzner box. The grant covers |
| 82 |
one script and nothing else; the script bounds its own arguments (source under |
| 83 |
`/var/tmp/bento-deploy`, destination under `/usr/local/bin`, a bare `*.service` |
| 84 |
unit), so it is a script-guarded grant rather than a broad |
| 85 |
`install` + `systemctl` one. |
| 86 |
|
| 87 |
## The test runner on astra |
| 88 |
|
| 89 |
astra is the only instance that runs test suites, and it runs them itself rather |
| 90 |
than over SSH. `pom.service` runs as the `pom` user with `ProtectHome`, so it can |
| 91 |
see neither max's checkouts nor max's toolchain, and astra has no sshd for a hop |
| 92 |
to reach (Tailscale SSH does not intercept a node connecting to itself, which is |
| 93 |
why the config's old `ssh = "max@<astra>"` failed `Connection refused` and no |
| 94 |
suite ever ran). `TestsConfig.ssh` is therefore optional; omitted, the command |
| 95 |
runs as a local child. |
| 96 |
|
| 97 |
Everything the runner needs lives under `/var/lib/pom`, the one path the hardened |
| 98 |
unit can write: |
| 99 |
|
| 100 |
|
| 101 |
|
| 102 |
| Runner script | `/var/lib/pom/staging/run-ci.sh` | `run-ci.sh` here | |
| 103 |
| Unit overrides | `/etc/systemd/system/pom.service.d/10-test-runner.conf` | `pom.service.d-10-test-runner.conf` here | |
| 104 |
| Clones | `/var/lib/pom/staging/{MNW,synckit,Apps/*}` | cloned from `/home/max/git-mirrors` | |
| 105 |
| Toolchain | `/var/lib/pom/.cargo`, `/var/lib/pom/.rustup` | rustup, as the `pom` user, plus `sqlx-cli` | |
| 106 |
|
| 107 |
The drop-in is not optional: the base unit's `MemoryMax=256M` would OOM-kill any |
| 108 |
cargo build, and `ProtectHome` has to be `read-only` rather than `true` so the |
| 109 |
clones can fetch from the mirrors. |
| 110 |
|
| 111 |
`staging/` mirrors the `~/Code` tree, `Apps/` included. Nothing in git records |
| 112 |
that layout, and both Tauri apps carry a symlink that reaches across it |
| 113 |
(`src-tauri/frontend/js/shared-updater.js` into `MNW/shared/tauri-updater-ui`). |
| 114 |
Flatten the clones and it dangles, and the app's `build.rs` panics on a read. |
| 115 |
|
| 116 |
Postgres notes, all of which cause failures that look like something else: |
| 117 |
`PGUSER=pom` is required because sqlx cannot resolve a username inside the |
| 118 |
sandbox and falls back to `whoami`'s `anonymous` placeholder; the harnesses need |
| 119 |
`TEST_DATABASE_URL` because they default to a TCP URL and astra's postgres is |
| 120 |
socket-only; and the MNW suite migrates its own `pom_ci_makenotwork` rather than |
| 121 |
compiling against the committed `.sqlx` cache, which goes stale silently whenever |
| 122 |
a migration lands without a `cargo sqlx prepare`. |
| 123 |
|
| 124 |
**The host timezone is load-bearing.** astra was on `America/Los_Angeles` and is |
| 125 |
now on `America/Denver`, matching fw13. goingson's |
| 126 |
`a_relative_event_keeps_its_wall_clock_across_a_move` asserts a civil time that |
| 127 |
only holds at UTC-6, and it reads `/etc/localtime` rather than `TZ`, so exporting |
| 128 |
`TZ` in this script does not move it — only the host setting does. That is a |
| 129 |
latent fragility in the test worth fixing at the source; until then a CI host on |
| 130 |
any other zone reports goingson red for no reason. |
| 131 |
|
| 132 |
To reproduce a red suite by hand: |
| 133 |
|
| 134 |
``` |
| 135 |
sudo -u pom /var/lib/pom/staging/run-ci.sh mnw |
| 136 |
``` |
| 137 |
|
| 138 |
A failure that only appears under `systemd-run` with the unit's properties is a |
| 139 |
hardening problem, not a test problem. |
| 140 |
|
| 141 |
## Rollback |
| 142 |
|
| 143 |
The installer keeps the previous binary as `<install-path>.prev`. Rolling back |
| 144 |
is putting it back and restarting: |
| 145 |
|
| 146 |
``` |
| 147 |
sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom |
| 148 |
sudo systemctl restart pom.service |
| 149 |
``` |
| 150 |
|