Skip to main content

max / makenotwork

pom: prepare a node for a Sando promote, and document the path that ships it
Author: Max Johnson <me@maxj.phd> · 2026-08-16 01:23 UTC
Signed with PGP, not checked
Commit: 18171491e19d546adec6e02841f170273c39a881
Parent: 5526b0d
3 files changed, +300 insertions, -40 deletions
M sando/pom-topology.toml +14 -2
@@ -55,7 +55,11 @@
55 55 [[tier.node]]
56 56 name = "astra"
57 57 platform = "linux/aarch64"
58 - ssh_target = "max@astra" # tailnet name
58 + # Tailnet name, reached over Tailscale SSH. The connection is made by the
59 + # `sando` user on fw13, so astra's host key has to be in /srv/sando/.ssh/
60 + # known_hosts; without it the deploy fails at `Host key verification failed`
61 + # before it has done anything, which is where it stood on 2026-08-15.
62 + ssh_target = "max@astra"
59 63 release_root = "/opt/pom"
60 64 service_name = "pom.service"
61 65 # Readiness on top of `systemctl is-active`: pom binds its API on the tailnet
@@ -75,7 +79,15 @@
75 79 [[tier.node]]
76 80 name = "hetzner"
77 81 platform = "linux/x86_64"
78 - ssh_target = "root@alpha-west-1" # Tailscale SSH; port 2200 via the ssh config
82 + # Not Tailscale SSH, whatever this line used to say. sandod runs as the `sando`
83 + # user on fw13 and that user's ~/.ssh/config maps `alpha-west-1` to port 2200,
84 + # which is the box's own sshd; Tailscale SSH is 22 and never sees the
85 + # connection. So this needs sando's public key in root's authorized_keys there,
86 + # and it is not in it today: measured 2026-08-15, `Permission denied
87 + # (publickey)`. mnw reaches the same machine as `makenotwork@alpha-west-1`, a
88 + # service user with the key and a scoped systemctl grant, which is the house
89 + # pattern if this one wants revisiting.
90 + ssh_target = "root@alpha-west-1"
79 91 release_root = "/opt/pom"
80 92 service_name = "pom.service"
81 93 health_url = "http://127.0.0.1:9100/api/health"
M pom/deploy/README.md +100 -38
@@ -1,13 +1,18 @@
1 1 # Deploying pom
2 2
3 - pom ships through **Bento**, not Sando, and not by hand.
3 + pom is **built by Bento and shipped by Sando**, and not by hand. Bento builds
4 + and packages; Sando decides whether a thing advances a stage, and performs the
5 + advance. Design: wiki `sando-bento-boundary`.
4 6
5 - Sando cannot do this job. `sando-daemon.toml` sets `build_host = "fw13"` and
6 - `build::run` refuses to compile anywhere else, which is the never-build-on-prod
7 - invariant and also makes Sando single-architecture. pom runs on astra (aarch64)
8 - and on the Hetzner box (x86_64), so Sando could never build half of this release
9 - without breaking its own rule. Bento already fans out native builds across both
10 - hosts, so pom is a Bento recipe.
7 + Neither controller could do this alone. `build::run` compiles only on the one
8 + host `build_host` names, which is the never-build-on-prod invariant and also
9 + makes Sando single-architecture; pom runs on astra (aarch64) and on the Hetzner
10 + box (x86_64), so Sando could never build half of a release without breaking its
11 + own rule. Bento fans out native builds across both machines and cannot honestly
12 + gate an artifact in a place it does not know about. So `pom/bento.toml` declares
13 + `kind = "service"` with no `[[deploy]]` tables, and the daemon's `[handoff.pom]`
14 + carries the collected binary to sandod, which owns install, restart, health and
15 + rollback from there.
11 16
12 17 `deploy.sh` is gone. It cross-compiled with `cargo zigbuild` (against the
13 18 native-per-architecture rule, with a cargo-zigbuild that is not installed on
@@ -17,34 +22,83 @@
17 22
18 23 ## Running a deploy
19 24
20 - Bump the version in `Cargo.toml`, commit, tag `v<version>`, push, then trigger
21 - the build through Bento for both targets. The `/deploy` skill has the procedure.
25 + Bump the version in `Cargo.toml`, commit, tag `pom-v<version>`, push, then
26 + trigger the build through Bento for both targets. The `/deploy` skill has the
27 + procedure.
22 28
23 - What the recipe does, per target (`dist/recipes/linux.rhai`):
29 + Bento's half, per target (`dist/recipes/linux.rhai`):
24 30
25 31 1. Pin the checkout to the release tag, and check both hosts report one commit.
26 32 2. `cargo clippy -D warnings` and `cargo test`, on that target's own build host.
27 33 3. `cargo build --release`.
28 - 4. Compare the binary's highest `GLIBC_` symbol against the service host's
29 - `ldd --version`, and assert `pom --version` matches the tag.
30 - 5. Stage the binary on the service host and call the privileged installer.
31 - 6. Poll `/api/health` until the restarted instance answers, then assert the
32 - running binary reports the version that was just installed.
34 + 4. Assert `pom --version` matches the tag.
35 + 5. `collect` the binary. The daemon hashes it, writes the artifact record beside
36 + it, stages it into `/srv/sando/pom/releases/staging` over ssh, and calls
37 + `POST /apps/pom/intake`.
33 38
34 - The two instances go one at a time, and `require_all_targets` keeps the release
35 - from counting as done until both are green. pom watches its own deploy, so the
36 - restart takes one watcher down for a moment; the other stays up and keeps
37 - watching. Never both at once.
39 + The recipe stops there. There is no `glibc_check` in it any more and no health
40 + poll: both are evidence about the artifact *in a place*, which a build host
41 + cannot honestly produce. They moved to Sando's side, where the node is known.
42 +
43 + Sando's half, one tier at a time (`sando/pom-topology.toml`):
44 +
45 + | Tier | Node | Gates |
46 + |------|------|-------|
47 + | `host` | none — the tier an intake lands on | none; Bento's verdicts ride in the artifact record |
48 + | `astra` | astra, aarch64 | `node_health`, `burn_in` 24h, `manual_confirm` |
49 + | `hetzner` | alpha-west-1, x86_64 | `node_health`, `manual_confirm` |
50 +
51 + ```
52 + curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
53 + "$SANDO_DAEMON/apps/pom/promote/astra"
54 + ```
55 +
56 + Note the `/apps/pom` prefix. The unprefixed routes address the default product,
57 + which is `mnw`; `POST /promote/astra` from muscle memory promotes the wrong
58 + thing. Read state the same way: `GET /apps/pom/status.json`.
59 +
60 + astra goes first because it is the instance that matters least if pom is down for
61 + a moment: it watches, it does not serve anyone. pom watches its own deploy, so
62 + the restart takes one watcher down; the Hetzner instance keeps watching, and
63 + health is read from the peer over the mesh. Never both at once.
64 +
65 + ## One-time node setup
66 +
67 + A node has to be prepared before its first promote, and the failure if it is not
68 + is silent rather than loud: sandod rsyncs the bundle, swaps `current`, restarts
69 + a unit still pointing at `/usr/local/bin/pom`, and `node_health` passes on the
70 + old binary. Sando reports the version shipped and nothing shipped.
71 +
72 + `bootstrap-pom-node.sh` is that preparation, idempotent, run on the node as root:
73 +
74 + ```
75 + sudo DEPLOY_USER=max \
76 + SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \
77 + ./bootstrap-pom-node.sh
78 + ```
79 +
80 + `DEPLOY_USER` is the user in that node's `ssh_target`: `max` on astra, `root` on
81 + the Hetzner box. It creates `/opt/pom` owned by that user, grants it
82 + `systemctl reload-or-restart pom.service` through one scoped sudoers line, seeds
83 + the currently-installed binary as the first release so the unit never points at a
84 + dangling symlink, and moves `ExecStart` to `/opt/pom/current/pom` through a
85 + drop-in. The node comes out on the version it went in on, deployable.
86 +
87 + It also has to be reachable: sandod runs as the `sando` user on fw13, so that
88 + user's ssh (its key, and its `known_hosts`) is what has to reach each node, not
89 + max's.
38 90
39 91 ## What a deploy does not touch
40 92
41 93 **Config.** `pom-astra.toml` and `pom-hetzner.toml` differ per instance, and the
42 94 live config on prod once carried a `[targets.mnw.tests]` block this repo did not
43 95 have. `deploy.sh` would have silently deleted it. Config is a separate,
44 - deliberate act; the installer refuses to go near it.
96 + deliberate act; nothing in the pipeline goes near it.
45 97
46 98 **The unit file.** Same reasoning. A hardened unit that has drifted from the
47 - repo is a question for a human, not something a binary deploy overwrites.
99 + repo is a question for a human, not something a binary deploy overwrites. That
100 + is why the bootstrap moves `ExecStart` in a drop-in: one reversible file beside
101 + the unit, rather than an edit of it.
48 102
49 103 ## The database path is config, not environment
50 104
@@ -66,23 +120,18 @@
66 120 sudo -u pom pom --init --config /etc/pom/pom.toml status
67 121 ```
68 122
69 - ## One-time host setup
123 + ## The retired Bento install path
70 124
71 - Each host that runs pom needs the installer and its scoped sudo grant:
125 + `install-service.sh` and `bento-deploy.sudoers` are what Bento used when it
126 + installed pom itself: a single script-guarded sudo grant, staging under
127 + `/var/tmp/bento-deploy`, atomic `install(1)` to `/usr/local/bin/pom` with the
128 + previous binary kept as `.prev`. Nothing in the pipeline calls them any more.
72 129
73 - ```
74 - sudo install -d /usr/local/lib/bento
75 - sudo install -m 0755 install-service.sh /usr/local/lib/bento/install-service.sh
76 - sudo install -m 0440 bento-deploy.sudoers /etc/sudoers.d/bento-deploy
77 - sudo visudo -cf /etc/sudoers.d/bento-deploy
78 - ```
79 -
80 - Edit the sudoers file first so the user matches how Bento reaches that host:
81 - `max` on astra, `root` over Tailscale SSH on the Hetzner box. The grant covers
82 - one script and nothing else; the script bounds its own arguments (source under
83 - `/var/tmp/bento-deploy`, destination under `/usr/local/bin`, a bare `*.service`
84 - unit), so it is a script-guarded grant rather than a broad
85 - `install` + `systemctl` one.
130 + They are still here, and still installed on both nodes, because until a Sando
131 + promote has actually landed on a node, `/usr/local/bin/pom` is what that node is
132 + running and `.prev` is its rollback. Remove both files, and the grant and
133 + installer on each host, in a separate pass once each node has taken a release
134 + through Sando.
86 135
87 136 ## The test runner on astra
88 137
@@ -140,8 +189,21 @@
140 189
141 190 ## Rollback
142 191
143 - The installer keeps the previous binary as `<install-path>.prev`. Rolling back
144 - is putting it back and restarting:
192 + Sando owns it, per tier:
193 +
194 + ```
195 + curl -H "Authorization: Bearer $SANDO_API_TOKEN" -X POST \
196 + "$SANDO_DAEMON/apps/pom/rollback/astra"
197 + ```
198 +
199 + That is the whole gain from the move. Release dirs are named for their content
200 + digest rather than their version, so every release the node has kept is a
201 + rollback target and the horizon is as deep as the GC allows. What pom had before
202 + was a single `<install-path>.prev` file: one version back, overwritten by the
203 + next deploy.
204 +
205 + A node that has not taken a Sando release yet is still on the old path, and
206 + rolling it back is putting `.prev` back by hand:
145 207
146 208 ```
147 209 sudo install -m 0755 /usr/local/bin/pom.prev /usr/local/bin/pom
A pom/deploy/bootstrap-pom-node.sh +186
@@ -1,0 +1,186 @@
1 + #!/usr/bin/env bash
2 + # Idempotent bootstrap for a pom node under Sando. Run on the node as root.
3 + #
4 + # pom used to be installed by Bento straight into /usr/local/bin/pom. Under the
5 + # Sando/Bento boundary Bento builds and hands off, and Sando deploys the way it
6 + # deploys everything else: a content-addressed bundle under <release_root>,
7 + # an atomic `current` symlink, and `systemctl reload-or-restart` on the unit.
8 + #
9 + # /opt/pom/
10 + # releases/<digest16>/{pom,MANIFEST}
11 + # current -> releases/<digest16>
12 + #
13 + # Three things have to be true on the node before the first promote, and none of
14 + # them were:
15 + #
16 + # 1. <release_root> exists and the user Sando SSHes in as can write it.
17 + # sandod runs `mkdir -p <release_root>/releases/<id>` as that user, and /opt
18 + # is root-owned.
19 + # 2. That user may restart the unit: sandod runs
20 + # `sudo /bin/systemctl reload-or-restart pom.service`.
21 + # 3. The unit's ExecStart points at <release_root>/current/<bin>. This is the
22 + # one that fails silently rather than loudly: a promote to a node whose unit
23 + # still runs /usr/local/bin/pom rsyncs the bundle, swaps the symlink,
24 + # restarts the service onto the OLD binary, and passes node_health. Sando
25 + # reports the version shipped and nothing shipped.
26 + #
27 + # The ExecStart move is a drop-in, not an edit of pom.service. A deploy never
28 + # rewrites the unit (deploy/README.md, "What a deploy does not touch"), and the
29 + # same restraint applies here: the override is one file, it is legible next to
30 + # the unit it changes, and removing it puts the node back.
31 + #
32 + # Ordering is why this is a script rather than a runbook. Repointing ExecStart
33 + # before a bundle exists leaves the unit pointing at a dangling symlink until the
34 + # first promote, and any restart in between (a reboot, Restart=on-failure) takes
35 + # pom down for however long that is. So the currently-installed binary is seeded
36 + # as the first release and `current` points at it BEFORE the unit moves. The node
37 + # comes out of this on the same version it went in on, deployable.
38 + #
39 + # Usage:
40 + # sudo DEPLOY_USER=max SANDO_PUBKEY="$(ssh fw13 'sudo cat /srv/sando/.ssh/id_ed25519.pub')" \
41 + # ./bootstrap-pom-node.sh
42 + #
43 + # Required env:
44 + # DEPLOY_USER — the user in the node's `ssh_target` in pom-topology.toml
45 + # (`max` on astra, `root` on the Hetzner box).
46 + #
47 + # Optional env:
48 + # SANDO_PUBKEY — sando's public key on the Sando host, appended to
49 + # DEPLOY_USER's authorized_keys. Omit on a host reached by
50 + # Tailscale SSH, which does not consult authorized_keys.
51 + # RELEASE_ROOT — defaults to /opt/pom. Must match the node's `release_root`.
52 + # BIN_NAME — defaults to pom. Must match sando-pom.toml's `bin_names[0]`.
53 + # SERVICE_NAME — defaults to pom.service.
54 + # CONFIG_PATH — defaults to /etc/pom/pom.toml. The unit's --config argument.
55 + # HEALTH_URL — defaults to http://127.0.0.1:9100/api/health.
56 + #
57 + # What this does NOT do, and will not:
58 + #
59 + # Config. pom-astra.toml and pom-hetzner.toml differ per instance and the live
60 + # copies have carried blocks this repo did not. Config is a deliberate act.
61 + #
62 + # Remove the old Bento install path. /usr/local/bin/pom, its `.prev`, the
63 + # /usr/local/lib/bento installer and its sudoers grant are all left in place:
64 + # until a promote has actually landed on this node, that binary is what the
65 + # rollback is. Clearing them is a separate pass after the cutover, and the
66 + # script prints the paths so they are not forgotten.
67 +
68 + set -euo pipefail
69 +
70 + if [[ $EUID -ne 0 ]]; then
71 + echo "must run as root" >&2
72 + exit 1
73 + fi
74 + if [[ -z "${DEPLOY_USER:-}" ]]; then
75 + echo "DEPLOY_USER env var is required (the user in this node's ssh_target)" >&2
76 + exit 1
77 + fi
78 + if ! id "$DEPLOY_USER" &>/dev/null; then
79 + echo "no such user: $DEPLOY_USER" >&2
80 + exit 1
81 + fi
82 +
83 + RELEASE_ROOT="${RELEASE_ROOT:-/opt/pom}"
84 + BIN_NAME="${BIN_NAME:-pom}"
85 + SERVICE_NAME="${SERVICE_NAME:-pom.service}"
86 + CONFIG_PATH="${CONFIG_PATH:-/etc/pom/pom.toml}"
87 + HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:9100/api/health}"
88 + DROPIN_DIR="/etc/systemd/system/${SERVICE_NAME}.d"
89 + DROPIN="${DROPIN_DIR}/20-release-root.conf"
90 +
91 + log() { echo "[bootstrap-pom] $*"; }
92 +
93 + log "1/6 release root at $RELEASE_ROOT, writable by $DEPLOY_USER"
94 + install -d -o "$DEPLOY_USER" -g "$(id -gn "$DEPLOY_USER")" -m 0755 \
95 + "$RELEASE_ROOT" "$RELEASE_ROOT/releases"
96 +
97 + log "2/6 seeding the running binary as a release, so the unit never points at nothing"
98 + if [[ -L "$RELEASE_ROOT/current" ]]; then
99 + log " current -> $(readlink "$RELEASE_ROOT/current") (already seeded, left alone)"
100 + else
101 + # Whatever the unit runs today, resolved from its own ExecStart rather than
102 + # assumed to be /usr/local/bin/pom.
103 + running="$(systemctl show -p ExecStart --value "$SERVICE_NAME" \
104 + | sed -n 's/.*path=\([^ ;]*\).*/\1/p' | head -1)"
105 + if [[ -z "$running" || ! -x "$running" ]]; then
106 + echo "cannot find the binary $SERVICE_NAME currently runs (ExecStart=$running)." >&2
107 + echo "Seed $RELEASE_ROOT/current by hand, or promote before repointing the unit." >&2
108 + exit 4
109 + fi
110 + ver="$("$running" --version 2>/dev/null | awk '{print $NF}')"
111 + seed="$RELEASE_ROOT/releases/preexisting-${ver:-unknown}"
112 + install -d -o "$DEPLOY_USER" -g "$(id -gn "$DEPLOY_USER")" -m 0755 "$seed"
113 + install -o "$DEPLOY_USER" -g "$(id -gn "$DEPLOY_USER")" -m 0755 "$running" "$seed/$BIN_NAME"
114 + # No MANIFEST: this bundle came from the node, not from an intake, and a
115 + # hand-written one would claim an artifact record that does not exist.
116 + # Sando skips verification for a bundle without one and says so in the log.
117 + ln -sfn "releases/preexisting-${ver:-unknown}" "$RELEASE_ROOT/current.new"
118 + mv -Tf "$RELEASE_ROOT/current.new" "$RELEASE_ROOT/current"
119 + chown -h "$DEPLOY_USER:$(id -gn "$DEPLOY_USER")" "$RELEASE_ROOT/current"
120 + log " seeded $running (${ver:-unknown}) -> $RELEASE_ROOT/current"
121 + fi
122 +
123 + log "3/6 sudo grant: $DEPLOY_USER may restart $SERVICE_NAME"
124 + if [[ "$DEPLOY_USER" == "root" ]]; then
125 + log " root needs no grant; skipping (sudo must still be installed)"
126 + command -v sudo >/dev/null || { echo "sudo is not installed; sandod shells out to it" >&2; exit 5; }
127 + else
128 + cat > "/etc/sudoers.d/${DEPLOY_USER}-pom" <<EOF
129 + $DEPLOY_USER ALL=(ALL) NOPASSWD: /bin/systemctl reload-or-restart $SERVICE_NAME, /bin/systemctl restart $SERVICE_NAME, /bin/systemctl status $SERVICE_NAME
130 + EOF
131 + chmod 0440 "/etc/sudoers.d/${DEPLOY_USER}-pom"
132 + visudo -c -f "/etc/sudoers.d/${DEPLOY_USER}-pom" >/dev/null
133 + fi
134 +
135 + log "4/6 sando's key for $DEPLOY_USER"
136 + if [[ -n "${SANDO_PUBKEY:-}" ]]; then
137 + home="$(getent passwd "$DEPLOY_USER" | cut -d: -f6)"
138 + install -d -o "$DEPLOY_USER" -g "$(id -gn "$DEPLOY_USER")" -m 0700 "$home/.ssh"
139 + if ! grep -qF "$SANDO_PUBKEY" "$home/.ssh/authorized_keys" 2>/dev/null; then
140 + echo "$SANDO_PUBKEY" >> "$home/.ssh/authorized_keys"
141 + fi
142 + chown "$DEPLOY_USER:$(id -gn "$DEPLOY_USER")" "$home/.ssh/authorized_keys"
143 + chmod 0600 "$home/.ssh/authorized_keys"
144 + else
145 + log " SANDO_PUBKEY unset; assuming Tailscale SSH reaches this node"
146 + fi
147 +
148 + log "5/6 ExecStart -> $RELEASE_ROOT/current/$BIN_NAME (drop-in, unit untouched)"
149 + install -d -m 0755 "$DROPIN_DIR"
150 + cat > "$DROPIN" <<EOF
151 + # Written by pom/deploy/bootstrap-pom-node.sh. Sando deploys pom as a
152 + # content-addressed bundle and swaps <release_root>/current; the unit has to
153 + # follow the symlink or a promote restarts the service onto the old binary and
154 + # still reports success.
155 + #
156 + # A drop-in rather than an edit of pom.service, so the release path is one
157 + # reversible file and the hardened unit stays the unit this repo ships.
158 + [Service]
159 + ExecStart=
160 + ExecStart=$RELEASE_ROOT/current/$BIN_NAME serve --config $CONFIG_PATH
161 + EOF
162 + systemctl daemon-reload
163 +
164 + log "6/6 restart and prove the node is serving from the new path"
165 + systemctl reload-or-restart "$SERVICE_NAME"
166 + for _ in $(seq 1 30); do
167 + if curl -fsS --max-time 3 "$HEALTH_URL" >/dev/null 2>&1; then
168 + served=1
169 + break
170 + fi
171 + sleep 1
172 + done
173 + if [[ "${served:-0}" != 1 ]]; then
174 + echo "$SERVICE_NAME did not answer $HEALTH_URL within 30s." >&2
175 + echo "Roll back by removing $DROPIN and running: systemctl daemon-reload && systemctl restart $SERVICE_NAME" >&2
176 + exit 6
177 + fi
178 +
179 + echo
180 + log "Done. $SERVICE_NAME is running $("$RELEASE_ROOT/current/$BIN_NAME" --version 2>/dev/null)"
181 + log "Sando can now promote to this node: POST /apps/pom/promote/<tier>"
182 + echo
183 + log "Left in place on purpose, until a promote has landed here:"
184 + log " /usr/local/bin/$BIN_NAME (+ .prev) the old Bento install path, and today's rollback"
185 + log " /usr/local/lib/bento/install-service.sh + /etc/sudoers.d/bento-deploy"
186 + log "Clear both in a separate pass once this node has taken a Sando release."