Skip to main content

max / makenotwork

Turn pom's release into a hand-off to Sando pom stops installing itself. The manifest drops its [[deploy]] tables and the recipe ends at collect, so the daemon's [handoff.pom] carries the artifact to sandod, which owns install, restart, health and rollback. What stays in the recipe is evidence about the artifact -- clippy, tests, the glibc check, the version assertion. What goes is evidence about the artifact in a place, which a build host cannot honestly produce.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-09 16:53 UTC
Signed with PGP, not checked
Commit: 395681c669d71669e896579719467d573b34b4a8
Parent: db132ec
2 files changed, +79 insertions, -63 deletions
M pom/bento.toml +43 -36
@@ -2,9 +2,23 @@
2 2 # versioned with the code it describes.
3 3 #
4 4 # A service, not an app or a library: pom is run rather than distributed. There
5 - # is no bundle for a user to download and no crate to publish, so the release
6 - # ends at `deploy` -- the binary lands on the hosts that run it and their units
7 - # restart.
5 + # is no bundle for a user to download and no crate to publish.
6 + #
7 + # The kind says what pom IS. Who delivers it is a separate question, and the
8 + # answer changed: Bento used to install the binary and restart the unit itself
9 + # (`[[deploy]]` tables, now gone). Under the Sando/Bento boundary Bento builds
10 + # and packages, and Sando decides whether a thing advances a stage. So the
11 + # release ends at `collect`, and the daemon's `[handoff.pom]` table carries the
12 + # artifact to sandod, which verifies it against its record, gates it, and
13 + # promotes it tier by tier.
14 + #
15 + # That is what pom gets out of the move: a content-addressed release history and
16 + # a real rollback horizon, instead of the single `<install-path>.prev` file
17 + # Bento's installer kept. pom had the worst deploy story of anything shipped
18 + # purely because of its architecture; this is the fix.
19 + #
20 + # Bento refuses to start if this file and the daemon config disagree — a service
21 + # must either deploy itself or hand off, never neither and never both.
8 22 kind = "service"
9 23
10 24 # Two arches, two machines, built natively on each. Sando cannot do this job:
@@ -23,40 +37,33 @@
23 37 # makes a bare `v0.4.1` ambiguous the day it is created rather than eventually.
24 38 tag_format = "pom-v{version}"
25 39
26 - # NOT the both-or-neither gate it looks like. `require_all_targets` is consulted
27 - # by `publish`, and a service never publishes -- it deploys. So this flag does
28 - # nothing here today, and the 0.4.1 release proved it: x86_64 installed on the
29 - # Hetzner box and restarted it while the aarch64 build was still failing its test
30 - # gate, leaving the mesh split across two versions until the retry landed.
40 + # Still not the both-or-neither gate it looks like: `require_all_targets` is
41 + # consulted by `publish`, and pom does not publish. The 0.4.1 release showed what
42 + # that costs -- x86_64 installed on the Hetzner box and restarted it while the
43 + # aarch64 build was still failing its test gate, leaving the mesh split across
44 + # two versions until the retry landed.
31 45 #
32 - # Left on because it is the right intent and costs nothing, but do not read it as
33 - # a guarantee. Wiring it into deploy is not a one-line change either: the gate
34 - # asks that every OTHER target already be green, which for sequential deploys
35 - # means the first one could never go. A real both-or-neither needs a barrier
36 - # between build and deploy across the matrix. Tracked as an infra problem.
46 + # What changes under the handoff is where the guarantee comes from. It is no
47 + # longer Bento's to give: each arch is handed over as its own artifact with its
48 + # own evidence, and Sando resolves BOTH bundles before it touches a node, so a
49 + # version missing its x86_64 half fails the promote whole rather than halfway.
50 + # The split-mesh failure moves from "possible, untracked" to "refused by the
51 + # promote".
52 + #
53 + # Left on because it is the right intent and costs nothing.
37 54 require_all_targets = true
38 55
39 - # Where each target lands. The recipe never names a machine -- it calls
40 - # deploy(), and the target it is already building for decides where that goes.
56 + # No `[[deploy]]` tables. They are what a service that installs ITSELF declares,
57 + # and pom no longer does -- the daemon's `[handoff.pom]` sends the collected
58 + # artifact to sandod instead, and sandod owns install, restart, health and
59 + # rollback from there.
41 60 #
42 - # install_path is /usr/local/bin/pom, which is what pom.service actually
43 - # ExecStarts and what both live hosts have today. Not /opt/pom/pom: that was
44 - # copied from mnw-cli's companion shape and matches nothing on either box.
45 -
46 - [[deploy]]
47 - target = "linux/x86_64"
48 - # Reached over Tailscale SSH on 22 as root. NOT port 2200 -- that is what the
49 - # old deploy.sh used and it refuses this machine's key for both root and max,
50 - # which is why the 0.4.0 deploy was done by hand.
51 - host = "root@100.120.174.96"
52 - install_path = "/usr/local/bin/pom"
53 - service = "pom.service"
54 - health_url = "http://100.120.174.96:9100/api/health"
55 -
56 - [[deploy]]
57 - target = "linux/aarch64"
58 - # astra builds this one and astra runs it, so the binary never leaves the box.
59 - host = "astra"
60 - install_path = "/usr/local/bin/pom"
61 - service = "pom.service"
62 - health_url = "http://100.106.221.39:9100/api/health"
61 + # Kept here as the record of what the old path was, because the addresses moved
62 + # to sando's topology rather than disappearing:
63 + #
64 + # linux/x86_64 root@100.120.174.96 (Tailscale SSH on 22, NOT 2200 -- 2200 is
65 + # what the retired deploy.sh used and it refuses this machine's
66 + # key for both root and max, which is why 0.4.0 was done by hand)
67 + # linux/aarch64 astra
68 + # both install_path /usr/local/bin/pom, unit pom.service,
69 + # health http://<host>:9100/api/health
M pom/dist/recipes/linux.rhai +36 -27
@@ -13,6 +13,18 @@
13 13 // machine's key, and copied config and the unit file over the live ones with no
14 14 // diff and no backup. It did not work as written when 0.4.0 went out; that
15 15 // deploy was done by hand.
16 + //
17 + // This recipe no longer installs anything. Under the Sando/Bento boundary Bento
18 + // builds and packages; Sando decides whether a thing advances a stage. So the
19 + // recipe ends at `collect`, and the daemon's `[handoff.pom]` carries the
20 + // artifact to sandod, which verifies it against its record and runs the gates
21 + // that are about the artifact IN AN ENVIRONMENT -- node health, burn-in, the
22 + // restart, and rollback if it goes wrong.
23 + //
24 + // The split is not arbitrary. What stays here is evidence about the ARTIFACT:
25 + // clippy, the tests, the glibc check, the version assertion. Those are answerable
26 + // on a build host. What left is evidence about the artifact in a place, which a
27 + // build host cannot honestly produce.
16 28
17 29 let h = build_host();
18 30 let v = version();
@@ -50,33 +62,30 @@
50 62 // reporting something else means the checkout and the release disagree.
51 63 sh_ok(h, binary + " --version | grep -qw " + v);
52 64
53 - step("deploy");
54 - // Ships the binary and restarts the unit, through the root installer the host
55 - // holds a scoped sudo grant for. Config is deliberately untouched:
56 - // pom-astra.toml and pom-hetzner.toml differ per instance, and prod's carried a
57 - // [targets.mnw.tests] block this repo did not have. A deploy that copies config
58 - // over the top is how that block gets silently deleted.
59 - log(deploy(binary));
60 -
61 - // pom watches its own deploy, so the restart above takes the watcher down with
62 - // it for a moment. The two instances are deployed one at a time (Bento runs one
63 - // target per host at a time, and require_all_targets keeps the release from
64 - // being called done until both are green), so the other one stays up and keeps
65 - // watching while this one comes back.
65 + step("collect");
66 + // Hand the binary to the daemon, which hashes it, writes the artifact record
67 + // beside it, and (per `[handoff.pom]`) stages it into sando and asks for an
68 + // intake. Nothing is installed or restarted from here.
66 69 //
67 - // Assert the restarted instance is actually serving rather than trusting
68 - // systemctl's opinion that the unit started: pom answering /api/health is what
69 - // "pom is up" means. Retries because a restart is not instant, and a single
70 - // immediate probe would just be a race.
71 - sh_ok(deploy_host(),
72 - "for i in $(seq 1 30); do " +
73 - "curl -fsS -o /dev/null --max-time 5 " + health_url() + " && exit 0; " +
74 - "sleep 2; done; " +
75 - "echo 'pom did not answer " + health_url() + " within 60s after restart'; " +
76 - "systemctl status " + service_name() + " --no-pager || true; exit 1");
70 + // One collect per target, and the two never collide: the daemon files each
71 + // under <dist_root>/pom/<version>/<target>/, so the aarch64 and x86_64 binaries
72 + // are siblings rather than one overwriting the other. Sando then resolves which
73 + // of the two each node takes from the platform on its own build record, so the
74 + // aarch64 binary cannot land on the Hetzner box even by mistake.
75 + collect(h, binary, "pom", v);
77 76
78 - // And that what is serving is what was just installed, rather than an old
79 - // process that survived the restart.
80 - sh_ok(deploy_host(), install_path() + " --version | grep -qw " + v);
77 + log("pom " + v + " (" + target() + ") built on " + h + " and handed off");
81 78
82 - log("pom " + v + " live on " + deploy_host() + " (" + target() + ")");
79 + // What used to follow, and where it went:
80 + //
81 + // deploy(binary) -> sando promotes, through the ladder
82 + // health poll after restart -> sando's node_health gate on each tier
83 + // install_path --version -> sando verifies the bundle digest ON THE NODE
84 + // after rsync and before the symlink swap, which is
85 + // a stronger check than re-running --version: it
86 + // proves the bytes, not just the number they print.
87 + //
88 + // The bootstrap caveat that lived here still holds, and is now sando's to
89 + // honour: pom watches its own deploy, so restarting it takes the watcher down
90 + // with it. The instances are promoted one tier at a time and health is read
91 + // from a peer over the mesh -- never both at once.