Skip to main content

max / makenotwork

Ship pom through Bento, retiring deploy.sh deploy.sh did not work as written when 0.4.0 went out; that deploy was done by hand. It cross-compiled with cargo zigbuild, against the native-per-architecture rule and with a cargo-zigbuild that is not installed on fw13; it reached Hetzner on port 2200, which refuses this machine's key for both root and max; and it copied config and the unit file over the live ones with no diff and no backup. Prod's config carried a [targets.mnw.tests] block this repo did not have, so running it would have silently deleted a live check. Bento builds both arches natively -- x86_64 on fw13, aarch64 on astra -- and installs each on the machine that runs it. The recipe pins the release tag, gates on clippy and tests, checks the binary's glibc against the service host's and that pom --version matches the tag, then stages and installs through the root wrapper and polls /api/health until the restarted instance answers. Config and the unit file are untouched by design. Three facts the design note had wrong, checked against both live hosts: the install path is /usr/local/bin/pom, which is what pom.service ExecStarts (not /opt/pom/pom, and Sando's install-companion.sh refuses a dst outside /opt, so pom needs its own wrapper); Hetzner is reached on 22 as root over Tailscale SSH; the unit is /etc/systemd/system/pom.service. pom had no --version until now, which both the recipe and the installer rely on, and which is how an operator confirms what landed. pom watches its own deploy, so the two instances go one at a time and require_all_targets keeps the release from counting as done until both are green. The other instance stays up and keeps watching.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-30 23:32 UTC
Signed with PGP, not checked
Commit: 5b9d14f8c89aae6bd41c5f6f53d23d0087abb24f
Parent: a481bd2
7 files changed, +332 insertions, -101 deletions
M pom/src/main.rs +1
@@ -16,6 +16,7 @@
16 16 #[derive(Parser)]
17 17 #[command(
18 18 name = "pom",
19 + version,
19 20 about = "Peace of Mind: health checks and test orchestration"
20 21 )]
21 22 struct Cli {
A pom/bento.toml +48
@@ -1,0 +1,48 @@
1 + # How Bento releases pom. Lives here rather than in the daemon's config so it is
2 + # versioned with the code it describes.
3 + #
4 + # A service, not an app or a library: pom is run rather than distributed. There
5 + # is no bundle for a user to download and no crate to publish, so the release
6 + # ends at `deploy` -- the binary lands on the hosts that run it and their units
7 + # restart.
8 + kind = "service"
9 +
10 + # Two arches, two machines, built natively on each. Sando cannot do this job:
11 + # sando-daemon.toml sets build_host = "fw13" and build::run refuses to compile
12 + # anywhere else, which is the never-build-on-prod invariant and also makes Sando
13 + # single-architecture. pom needs an aarch64 binary for astra, so Sando could
14 + # never build half of this release without breaking its own rule.
15 + targets = ["linux/x86_64", "linux/aarch64"]
16 +
17 + # Workspace root carries the version.
18 + version_path = "Cargo.toml"
19 +
20 + # Neither instance ships without the other. pom watches production, and an
21 + # aarch64 build that quietly failed while x86_64 shipped leaves half the mesh on
22 + # an old binary with nothing saying so.
23 + require_all_targets = true
24 +
25 + # Where each target lands. The recipe never names a machine -- it calls
26 + # deploy(), and the target it is already building for decides where that goes.
27 + #
28 + # install_path is /usr/local/bin/pom, which is what pom.service actually
29 + # ExecStarts and what both live hosts have today. Not /opt/pom/pom: that was
30 + # copied from mnw-cli's companion shape and matches nothing on either box.
31 +
32 + [[deploy]]
33 + target = "linux/x86_64"
34 + # Reached over Tailscale SSH on 22 as root. NOT port 2200 -- that is what the
35 + # old deploy.sh used and it refuses this machine's key for both root and max,
36 + # which is why the 0.4.0 deploy was done by hand.
37 + host = "root@100.120.174.96"
38 + install_path = "/usr/local/bin/pom"
39 + service = "pom.service"
40 + health_url = "http://100.120.174.96:9100/api/health"
41 +
42 + [[deploy]]
43 + target = "linux/aarch64"
44 + # astra builds this one and astra runs it, so the binary never leaves the box.
45 + host = "astra"
46 + install_path = "/usr/local/bin/pom"
47 + service = "pom.service"
48 + health_url = "http://100.106.221.39:9100/api/health"
A pom/deploy/README.md +75
@@ -1,0 +1,75 @@
1 + # Deploying pom
2 +
3 + pom ships through **Bento**, not Sando, and not by hand.
4 +
5 + Sando cannot do this job. `sando-daemon.toml` sets `build_host = "fw13"` and
6 + `build::run` refuses to compile anywhere else, which is the never-build-on-prod
7 + invariant and also makes Sando single-architecture. pom runs on astra (aarch64)
8 + and on the Hetzner box (x86_64), so Sando could never build half of this release
9 + without breaking its own rule. Bento already fans out native builds across both
10 + hosts, so pom is a Bento recipe.
11 +
12 + `deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the
13 + native-per-architecture rule, with a cargo-zigbuild that is not installed on
14 + fw13), reached Hetzner on port 2200 which refuses this machine's key, and copied
15 + config and the unit file over the live ones with no diff and no backup. It did
16 + not work as written when 0.4.0 went out; that deploy was done by hand.
17 +
18 + ## Running a deploy
19 +
20 + Bump the version in `Cargo.toml`, commit, tag `v<version>`, push, then trigger
21 + the build through Bento for both targets. The `/deploy` skill has the procedure.
22 +
23 + What the recipe does, per target (`dist/recipes/linux.rhai`):
24 +
25 + 1. Pin the checkout to the release tag, and check both hosts report one commit.
26 + 2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host.
27 + 3. `cargo build --release`.
28 + 4. Compare the binary's highest `GLIBC_` symbol against the service host's
29 + `ldd --version`, and assert `pom --version` matches the tag.
30 + 5. Stage the binary on the service host and call the privileged installer.
31 + 6. Poll `/api/health` until the restarted instance answers, then assert the
32 + running binary reports the version that was just installed.
33 +
34 + The two instances go one at a time, and `require_all_targets` keeps the release
35 + from counting as done until both are green. pom watches its own deploy, so the
36 + restart takes one watcher down for a moment; the other stays up and keeps
37 + watching. Never both at once.
38 +
39 + ## What a deploy does not touch
40 +
41 + **Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the
42 + live config on prod once carried a `[targets.mnw.tests]` block this repo did not
43 + have. `deploy.sh` would have silently deleted it. Config is a separate,
44 + deliberate act; the installer refuses to go near it.
45 +
46 + **The unit file.** Same reasoning. A hardened unit that has drifted from the
47 + repo is a question for a human, not something a binary deploy overwrites.
48 +
49 + ## One-time host setup
50 +
51 + Each host that runs pom needs the installer and its scoped sudo grant:
52 +
53 + ```
54 + sudo install -d /usr/local/lib/bento
55 + sudo install -m 0755 install-service.sh /usr/local/lib/bento/install-service.sh
56 + sudo install -m 0440 bento-deploy.sudoers /etc/sudoers.d/bento-deploy
57 + sudo visudo -cf /etc/sudoers.d/bento-deploy
58 + ```
59 +
60 + Edit the sudoers file first so the user matches how Bento reaches that host:
61 + `max` on astra, `root` over Tailscale SSH on the Hetzner box. The grant covers
62 + one script and nothing else; the script bounds its own arguments (source under
63 + `/var/tmp/bento-deploy`, destination under `/usr/local/bin`, a bare `*.service`
64 + unit), so it is a script-guarded grant rather than a broad
65 + `install` + `systemctl` one.
66 +
67 + ## Rollback
68 +
69 + The installer keeps the previous binary as `<install-path>.prev`. Rolling back
70 + is putting it back and restarting:
71 +
72 + ```
73 + sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom
74 + sudo systemctl restart pom.service
75 + ```
A pom/deploy/bento-deploy.sudoers +21
@@ -1,0 +1,21 @@
1 + # Bento service-deploy grant. Install on each host that runs a Bento-deployed
2 + # service (currently astra and alpha-west-1, both running pom):
3 + #
4 + # sudo install -m 0440 bento-deploy.sudoers /etc/sudoers.d/bento-deploy
5 + # sudo visudo -cf /etc/sudoers.d/bento-deploy # validate before trusting it
6 + #
7 + # The deploy user -- the same user Bento's executor SSHes as -- may run ONLY the
8 + # service installer, with any arguments. The arguments are bounded by the script
9 + # itself (src under /var/tmp/bento-deploy, dst under /usr/local/bin, service a
10 + # bare *.service unit), so this is a script-guarded grant rather than a broad
11 + # install + systemctl one. Sando's mnw-companion.sudoers is the same line for
12 + # the same reason.
13 + #
14 + # Uncomment the line matching how Bento reaches this host. Both are listed
15 + # because pom's two instances are reached as different users: astra over the
16 + # tailnet as max, and the Hetzner box over Tailscale SSH as root. A root deploy
17 + # user does not need the grant at all -- the line is harmless there and keeps
18 + # the two hosts configured identically.
19 +
20 + max ALL=(root) NOPASSWD: /usr/local/lib/bento/install-service.sh *
21 + # root ALL=(root) NOPASSWD: /usr/local/lib/bento/install-service.sh *
D pom/deploy/deploy.sh -101
@@ -1,101 +1,0 @@
1 - #!/usr/bin/env bash
2 - #
3 - # STALE. Did not work as written when pom 0.4.0 went out on 2026-07-29, and the
4 - # 0.4.0 deploy was done by hand instead. Read this before trusting it.
5 - #
6 - # Three problems, in order of how much they matter:
7 - #
8 - # 1. It cross-compiles, which the ecosystem rule forbids: builds are native per
9 - # architecture. `cargo zigbuild --target aarch64-...` from fw13 is a cross
10 - # build for astra, and cargo-zigbuild is not installed here anyway (only zig
11 - # itself). What 0.4.0 actually did: build on astra for astra, build on fw13
12 - # for hetzner. Both ends are now x86_64 glibc 2.39 for the hetzner leg, so
13 - # that one needs nothing but `cargo build --release`; confirm with
14 - # `objdump -T target/release/pom | grep -o 'GLIBC_[0-9.]*' | sort -uV | tail -1`
15 - # against the target's `ldd --version` before shipping.
16 - # 2. It reaches hetzner on port 2200, which refused this machine's key for both
17 - # root and max. The route that worked is Tailscale SSH on 22 as root.
18 - # 3. It copies pom-<host>.toml over /etc/pom/pom.toml and pom.service over the
19 - # unit, with no diff and no backup. That is how a live-only config block gets
20 - # deleted: prod's config carried a [targets.mnw.tests] block this repo did
21 - # not have, so running this would have silently dropped prod's remote CI
22 - # check. Fixed in MNW@f92eb501, but the hazard is structural, not that one
23 - # block. Diff live against the file before installing either.
24 - #
25 - # What 0.4.0 did instead, per host: back up the binary, /etc/pom/pom.toml, and
26 - # pom.db + WAL; `systemctl stop pom`; install the binary; start; then confirm
27 - # migrations in the journal and that /api/health, /status.json and /api/versions
28 - # all answer 200.
29 - #
30 - set -euo pipefail
31 -
32 - ASTRA_HOST="max@100.106.221.39"
33 - HETZNER_HOST="root@100.120.174.96"
34 -
35 - SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
36 - PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
37 -
38 - deploy_target() {
39 - local name="$1"
40 - local host target sudo_prefix="" ssh_opts="" scp_opts=""
41 -
42 - case "$name" in
43 - astra)
44 - host="$ASTRA_HOST"
45 - target="aarch64-unknown-linux-gnu"
46 - sudo_prefix="sudo"
47 - ;;
48 - hetzner)
49 - host="$HETZNER_HOST"
50 - target="x86_64-unknown-linux-gnu"
51 - ssh_opts="-p 2200"
52 - scp_opts="-P 2200"
53 - ;;
54 - *)
55 - echo "Unknown target: $name (use astra, hetzner, or all)"
56 - exit 1
57 - ;;
58 - esac
59 -
60 - echo "=== Building pom for $name ($target) ==="
61 - cargo zigbuild --release --target "$target" --manifest-path "$PROJECT_DIR/Cargo.toml"
62 -
63 - local binary="$PROJECT_DIR/target/$target/release/pom"
64 -
65 - local config_file="$SCRIPT_DIR/pom-${name}.toml"
66 - if [ ! -f "$config_file" ]; then
67 - echo "Config not found: $config_file"
68 - exit 1
69 - fi
70 -
71 - echo "=== Deploying to $name ($host) ==="
72 - ssh $ssh_opts "$host" "$sudo_prefix mkdir -p /etc/pom"
73 - scp $scp_opts "$binary" "$host:/tmp/pom"
74 - scp $scp_opts "$config_file" "$host:/tmp/pom.toml"
75 - scp $scp_opts "$SCRIPT_DIR/pom.service" "$host:/tmp/pom.service"
76 -
77 - ssh $ssh_opts "$host" "$sudo_prefix mv /tmp/pom /usr/local/bin/pom && $sudo_prefix chmod +x /usr/local/bin/pom && $sudo_prefix mv /tmp/pom.toml /etc/pom/pom.toml && $sudo_prefix mv /tmp/pom.service /etc/systemd/system/pom.service"
78 -
79 - # Create env file with Postmark token if it doesn't exist
80 - ssh $ssh_opts "$host" "if [ ! -f /etc/pom/env ]; then echo 'POM_POSTMARK_TOKEN=SET_ME' | $sudo_prefix tee /etc/pom/env > /dev/null && $sudo_prefix chmod 600 /etc/pom/env; fi"
81 -
82 - ssh $ssh_opts "$host" "$sudo_prefix systemctl daemon-reload && $sudo_prefix systemctl enable pom && $sudo_prefix systemctl restart pom"
83 -
84 - echo "=== $name: deployed ==="
85 - ssh $ssh_opts "$host" "$sudo_prefix systemctl status pom --no-pager"
86 - }
87 -
88 - if [ $# -eq 0 ]; then
89 - echo "Usage: $0 <astra|hetzner|all>"
90 - exit 1
91 - fi
92 -
93 - case "$1" in
94 - all)
95 - deploy_target astra
96 - deploy_target hetzner
97 - ;;
98 - *)
99 - deploy_target "$1"
100 - ;;
101 - esac
A pom/deploy/install-service.sh +105
@@ -1,0 +1,105 @@
1 + #!/usr/bin/env bash
2 + # Install a staged service binary and restart its unit. Runs as ROOT, invoked by
3 + # Bento's deploy step over the host's executor as:
4 + #
5 + # sudo /usr/local/lib/bento/install-service.sh <src> <dst> <service>
6 + #
7 + # where <src> is the binary Bento staged under /var/tmp/bento-deploy/<app>/,
8 + # <dst> is the unit's ExecStart path (/usr/local/bin/pom), and <service> is the
9 + # systemd unit to restart (pom.service).
10 + #
11 + # This wrapper exists so the deploy user's sudo grant is ONE auditable script
12 + # rather than a broad install/systemctl grant on a box running production. Same
13 + # shape, and the same reasoning, as sando/deploy/install-companion.sh.
14 + #
15 + # Install (one-time per host, as root):
16 + # sudo install -d /usr/local/lib/bento
17 + # sudo install -m 0755 install-service.sh /usr/local/lib/bento/install-service.sh
18 + # # then add the scoped sudoers line -- see bento-deploy.sudoers
19 + #
20 + # What this deliberately does NOT touch:
21 + #
22 + # Config. pom-astra.toml and pom-hetzner.toml differ per instance, and the
23 + # live config on prod once carried a [targets.mnw.tests] block the repo did
24 + # not have. The old deploy.sh copied config over the top with no diff and no
25 + # backup, which is how that block would have been silently deleted. A deploy
26 + # ships the binary; config is a separate, deliberate act.
27 + #
28 + # The unit file. Same reasoning: a hardened unit that drifted from the repo is
29 + # a question for a human, not something a binary deploy overwrites.
30 + #
31 + # The install is atomic (install(1) writes a temp then renames), so a running
32 + # service never execs a half-written file. The previous binary is kept next to
33 + # the new one as <dst>.prev, which is the whole rollback: put it back and
34 + # restart. The unit is restarted only after a successful install, so a failed
35 + # install leaves the service on the binary it already had.
36 + set -euo pipefail
37 +
38 + if [[ $# -ne 3 ]]; then
39 + echo "usage: install-service.sh <src-binary> <dst-path> <service>" >&2
40 + exit 2
41 + fi
42 +
43 + STAGING_ROOT=/var/tmp/bento-deploy
44 +
45 + SRC="$1"
46 + DST="$2"
47 + SERVICE="$3"
48 +
49 + # Guard rails. These bound what a caller -- already behind the single sudoers
50 + # grant -- can install and restart. Being the ONLY control on a NOPASSWD grant,
51 + # they have to actually hold, so normalise BEFORE testing any prefix: a
52 + # glob-only test against the raw argument is something `..` walks straight out
53 + # of. `/var/tmp/bento-deploy/../../../etc/shadow` matches a naive src pattern,
54 + # and `/usr/local/bin/../../../etc/systemd/system/x` a naive dst one -- i.e.
55 + # `install -m 0755` as root to anywhere on the filesystem. `realpath -m`
56 + # resolves `..` and symlinks without requiring the path to exist (the dst does
57 + # not on a first install). The roots are resolved too, so the comparison still
58 + # holds if either is a symlink.
59 + SRC="$(realpath -m -- "$SRC")"
60 + DST="$(realpath -m -- "$DST")"
61 + STAGING_ROOT="$(realpath -m -- "$STAGING_ROOT")"
62 + BIN_ROOT="$(realpath -m -- /usr/local/bin)"
63 +
64 + case "$SRC" in
65 + "$STAGING_ROOT"/*) : ;;
66 + *) echo "install-service: refusing src outside $STAGING_ROOT: $SRC" >&2; exit 3 ;;
67 + esac
68 + case "$DST" in
69 + "$BIN_ROOT"/*) : ;;
70 + *) echo "install-service: refusing dst outside $BIN_ROOT: $DST" >&2; exit 3 ;;
71 + esac
72 + case "$SERVICE" in
73 + *[/[:space:]]*|"") echo "install-service: bad service name: $SERVICE" >&2; exit 3 ;;
74 + *.service) : ;;
75 + *) echo "install-service: service must end in .service: $SERVICE" >&2; exit 3 ;;
76 + esac
77 +
78 + if [[ ! -f "$SRC" ]]; then
79 + echo "install-service: no binary at $SRC" >&2
80 + exit 4
81 + fi
82 +
83 + # Refuse a binary that cannot run here. Bento checks this too, before it stages
84 + # anything, but that check reads the build host's objdump output; this one is
85 + # the target actually trying to load it. A unit restarted onto a binary it
86 + # cannot exec is down until someone notices.
87 + if ! "$SRC" --version >/dev/null 2>&1; then
88 + echo "install-service: staged binary at $SRC will not run on this host" >&2
89 + "$SRC" --version || true
90 + exit 5
91 + fi
92 +
93 + if [[ -f "$DST" ]]; then
94 + echo "install-service: keeping the current binary as $DST.prev"
95 + cp -p -- "$DST" "$DST.prev"
96 + fi
97 +
98 + # Installs the normalised paths, not the raw arguments, so there is no gap
99 + # between what was checked and what is written.
100 + echo "install-service: installing $SRC -> $DST"
101 + install -m 0755 -- "$SRC" "$DST"
102 +
103 + echo "install-service: restarting $SERVICE"
104 + systemctl restart "$SERVICE"
105 + echo "install-service: done ($SERVICE live on $(basename -- "$DST"))"
A pom/dist/recipes/linux.rhai +82
@@ -1,0 +1,82 @@
1 + // pom: Linux release recipe for Bento (x86_64 on fw13, aarch64 on astra).
2 + //
3 + // One recipe serves both arches. build_host() resolves to whichever native host
4 + // the topology assigns the target and deploy() resolves to whichever machine
5 + // bento.toml says that target runs on, so there is no cross-compilation and no
6 + // hard-coded host name anywhere in here. That matters more than usual: the two
7 + // legs of this release are different architectures on different machines, and a
8 + // recipe that named hosts could install the aarch64 binary on the x86_64 box.
9 + //
10 + // Replaces deploy/deploy.sh, which cross-compiled with cargo zigbuild (against
11 + // the native-per-architecture rule, and with a cargo-zigbuild that is not
12 + // installed on fw13 anyway), reached Hetzner on a port that refuses this
13 + // machine's key, and copied config and the unit file over the live ones with no
14 + // diff and no backup. It did not work as written when 0.4.0 went out; that
15 + // deploy was done by hand.
16 +
17 + let h = build_host();
18 + let v = version();
19 + let r = repo();
20 +
21 + step("checkout");
22 + // Pin to the release tag v<version>, not whatever main is at pull time. The
23 + // daemon runs the same pin as a cross-host barrier before either target builds,
24 + // so both arches come from one commit rather than from two machines' idea of
25 + // main.
26 + let sha = checkout_sha(h);
27 + log("pinned " + h + " to v" + v + " @ " + sha);
28 +
29 + // Gate: nothing is installed on a host that watches production from code that
30 + // fails clippy or its tests. It runs on this target's own build host, so a
31 + // break confined to one architecture is caught where it would have shipped.
32 + step("prebuild");
33 + sh_ok(h, "cd " + r + " && cargo clippy --workspace --all-targets " + feature_flags() + " -- -D warnings");
34 + sh_ok(h, "cd " + r + " && cargo test --workspace " + feature_flags());
35 +
36 + step("build");
37 + sh_ok(h, "cd " + r + " && cargo build --release " + feature_flags());
38 + let binary = resolve_artifact(h, r + "/target/release/pom");
39 +
40 + step("verify");
41 + // Native-per-arch removed the cross-compile hazard the old script was written
42 + // against, but not this one: fw13 tracks a newer glibc than the Ubuntu 24.04
43 + // box in Hetzner, so a binary built here can reference a symbol version that
44 + // box does not have and fail at exec -- after the unit has already restarted
45 + // onto it. This compares the two and fails the step if the build host got
46 + // ahead. The installer checks the same thing again on the far side, by actually
47 + // running the binary.
48 + log(glibc_check(binary));
49 + // The version that is about to ship is the version in the tag. A binary
50 + // reporting something else means the checkout and the release disagree.
51 + sh_ok(h, binary + " --version | grep -qw " + v);
52 +
53 + step("deploy");
54 + // Ships the binary and restarts the unit, through the root installer the host
55 + // holds a scoped sudo grant for. Config is deliberately untouched:
56 + // pom-astra.toml and pom-hetzner.toml differ per instance, and prod's carried a
57 + // [targets.mnw.tests] block this repo did not have. A deploy that copies config
58 + // over the top is how that block gets silently deleted.
59 + log(deploy(binary));
60 +
61 + // pom watches its own deploy, so the restart above takes the watcher down with
62 + // it for a moment. The two instances are deployed one at a time (Bento runs one
63 + // target per host at a time, and require_all_targets keeps the release from
64 + // being called done until both are green), so the other one stays up and keeps
65 + // watching while this one comes back.
66 + //
67 + // Assert the restarted instance is actually serving rather than trusting
68 + // systemctl's opinion that the unit started: pom answering /api/health is what
69 + // "pom is up" means. Retries because a restart is not instant, and a single
70 + // immediate probe would just be a race.
71 + sh_ok(deploy_host(),
72 + "for i in $(seq 1 30); do " +
73 + "curl -fsS -o /dev/null --max-time 5 " + health_url() + " && exit 0; " +
74 + "sleep 2; done; " +
75 + "echo 'pom did not answer " + health_url() + " within 60s after restart'; " +
76 + "systemctl status " + service_name() + " --no-pager || true; exit 1");
77 +
78 + // And that what is serving is what was just installed, rather than an old
79 + // process that survived the restart.
80 + sh_ok(deploy_host(), install_path() + " --version | grep -qw " + v);
81 +
82 + log("pom " + v + " live on " + deploy_host() + " (" + target() + ")");