Skip to main content

max / makenotwork

Give wam a release pipeline, and a --version to be deployed by The binary running on prod was an April build that predated wam's own require_auth middleware, so /tickets answered anonymous reads with the full ticket body. Nothing noticed for four months because a service placed by hand has no release history to be stale against. wam keeps [[deploy]] tables rather than handing off to Sando like pom does. The difference is blast radius, not house style: pom's bad deploy takes out the monitoring, wam's takes out an internal ticket list whose rollback is the .prev binary and a restart. --version is load-bearing now: install-service.sh runs it on the service host before installing, so a binary without it cannot deploy at all.
Author: Max Johnson <me@maxj.phd> · 2026-08-15 21:34 UTC
Signed with PGP, not checked
Commit: d2e908a88f516a7b8751a458869522b593cd4f1f
Parent: 849c337
5 files changed, +194 insertions, -8 deletions
M wam/Cargo.lock +42 -6
@@ -1721,7 +1721,7 @@
1721 1721
1722 1722 [[package]]
1723 1723 name = "wam"
1724 - version = "0.2.0"
1724 + version = "0.3.0"
1725 1725 dependencies = [
1726 1726 "axum",
1727 1727 "chrono",
@@ -2219,22 +2219,58 @@
2219 2219 source = "registry+https://github.com/rust-lang/crates.io-index"
2220 2220 checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
2221 2221
2222 + [[patch.unused]]
2223 + name = "synckit-client"
2224 + version = "0.8.0"
2225 +
2226 + [[patch.unused]]
2227 + name = "synckit-config"
2228 + version = "0.2.0"
2229 +
2222 2230 [[patch.unused]]
2223 2231 name = "docengine"
2224 - version = "0.4.0"
2232 + version = "0.7.0"
2225 2233
2226 2234 [[patch.unused]]
2227 2235 name = "kberg"
2228 2236 version = "0.1.0"
2229 2237
2238 + [[patch.unused]]
2239 + name = "ops-status"
2240 + version = "0.1.0"
2241 +
2230 2242 [[patch.unused]]
2231 2243 name = "tagtree"
2232 2244 version = "0.4.0"
2233 2245
2234 2246 [[patch.unused]]
2235 - name = "synckit-client"
2236 - version = "0.8.0"
2247 + name = "quasi-axum"
2248 + version = "0.11.0"
2237 2249
2238 2250 [[patch.unused]]
2239 - name = "synckit-config"
2240 - version = "0.2.0"
2251 + name = "quasi-basics"
2252 + version = "0.11.0"
2253 +
2254 + [[patch.unused]]
2255 + name = "quasi-http"
2256 + version = "0.11.0"
2257 +
2258 + [[patch.unused]]
2259 + name = "quasi-immediate"
2260 + version = "0.11.0"
2261 +
2262 + [[patch.unused]]
2263 + name = "quasi-router"
2264 + version = "0.11.0"
2265 +
2266 + [[patch.unused]]
2267 + name = "quasi-store"
2268 + version = "0.1.0"
2269 +
2270 + [[patch.unused]]
2271 + name = "quasi-tauri"
2272 + version = "0.11.0"
2273 +
2274 + [[patch.unused]]
2275 + name = "quasi-webview"
2276 + version = "0.11.0"
M wam/Cargo.toml +1 -1
@@ -1,6 +1,6 @@
1 1 [package]
2 2 name = "wam"
3 - version = "0.2.0"
3 + version = "0.3.0"
4 4 edition = "2024"
5 5 license = "MIT"
6 6
M wam/src/cli.rs +10 -1
@@ -4,8 +4,17 @@
4 4
5 5 use crate::types::{Channel, Priority, Status};
6 6
7 + /// `version` is not decoration: Bento's privileged installer runs
8 + /// `<staged-binary> --version` on the service host before it installs anything,
9 + /// so a binary without the flag cannot be deployed through the pipeline at all.
10 + /// It is also the check that catches an aarch64 build staged onto the x86_64
11 + /// box, which is the failure the two-arch fan-out exists to risk.
7 12 #[derive(Parser)]
8 - #[command(name = "wam", about = "Whack-a-Mole -- distributed ticket manager")]
13 + #[command(
14 + name = "wam",
15 + version,
16 + about = "Whack-a-Mole -- distributed ticket manager"
17 + )]
9 18 pub(crate) struct Cli {
10 19 #[command(subcommand)]
11 20 pub command: Option<Command>,
A wam/bento.toml +71
@@ -1,0 +1,71 @@
1 + # How Bento releases wam. Lives here rather than in the daemon's config so it is
2 + # versioned with the code it describes.
3 + #
4 + # A service, not an app or a library: wam is run rather than distributed. There
5 + # is no bundle for a user to download and no crate to publish.
6 + #
7 + # wam deploys ITSELF, and that is the deliberate difference from pom. pom hands
8 + # its artifact to Sando, which gates it and promotes it tier by tier; wam keeps
9 + # the `[[deploy]]` tables and lets Bento install and restart directly. The two
10 + # are not disagreeing about what good practice is. They are answering a question
11 + # about blast radius, and getting different answers:
12 + #
13 + # What does a bad deploy cost? For the MNW server it costs paying creators
14 + # their platform, so it earns burn-in, tiers, migration dry-runs and a
15 + # rollback horizon. For pom it costs the monitoring, which is how anyone finds
16 + # out the server broke, so it earns the gates too. For wam it costs an
17 + # internal ticket list nobody's money or morning depends on -- and the
18 + # recovery is `install-service.sh`'s own `<dst>.prev` plus a restart.
19 + #
20 + # Charging wam the full Sando ceremony would buy a rollback horizon for tickets
21 + # we could re-sync from the peer, and would cost an afternoon of topology per
22 + # release path. The ladder is a real cost and it should be spent where a bad
23 + # deploy is expensive. See wiki `deploy-ceremony-tiers`.
24 + kind = "service"
25 +
26 + # Two arches, two machines, built natively on each: x86_64 on fw13 for the
27 + # Hetzner box, aarch64 on astra for astra. Same reason pom needs both -- the
28 + # mesh has a node on each -- and the same reason neither can be built by Sando,
29 + # whose build::run compiles only on its single configured build host.
30 + targets = ["linux/x86_64", "linux/aarch64"]
31 +
32 + version_path = "Cargo.toml"
33 +
34 + # MNW is one .git over the server, sando, multithreaded, pom, wam and more, each
35 + # versioned separately, so the default `v{version}` names no product in
36 + # particular here.
37 + tag_format = "wam-v{version}"
38 +
39 + # Both nodes or neither. The mesh syncs tickets between them and the sync
40 + # protocol is the thing most likely to break across a version boundary, so a
41 + # release that lands on one node and fails on the other leaves the mesh talking
42 + # to itself across a version gap. That is exactly the split-mesh failure pom's
43 + # 0.4.1 release hit.
44 + require_all_targets = true
45 +
46 + # The service hosts. A deploy destination is deliberately NOT a `[[host]]` in
47 + # the daemon config: a build host is granted build/package, a service host
48 + # deploy/restart, and neither is granted the other's.
49 + #
50 + # install_path is /usr/local/bin/wam on both, which is a move: the running
51 + # binary is at /opt/wam/wam on Hetzner today, placed there by hand in April.
52 + # install-service.sh refuses any dst outside /usr/local/bin, and that guard is
53 + # the only thing bounding a NOPASSWD sudo grant, so the binary moves rather than
54 + # the guard widening. Both units' ExecStart move with it.
55 +
56 + # Hetzner, reached over Tailscale SSH on 22 as root. NOT port 2200: that is what
57 + # the retired deploy.sh used, and it refuses this machine's key.
58 + [[deploy]]
59 + target = "linux/x86_64"
60 + host = "root@100.120.174.96"
61 + install_path = "/usr/local/bin/wam"
62 + service = "wam.service"
63 + health_url = "http://127.0.0.1:7890/tickets"
64 +
65 + # astra, over the tailnet as max.
66 + [[deploy]]
67 + target = "linux/aarch64"
68 + host = "astra"
69 + install_path = "/usr/local/bin/wam"
70 + service = "wam.service"
71 + health_url = "http://127.0.0.1:7890/tickets"
A wam/dist/recipes/linux.rhai +70
@@ -1,0 +1,70 @@
1 + // wam: Linux release recipe for Bento (x86_64 on fw13, aarch64 on astra).
2 + //
3 + // One recipe serves both arches and both service hosts. build_host() resolves
4 + // to whichever native host the topology assigns the target, and deploy()
5 + // resolves to whichever machine bento.toml says that target runs on, so there
6 + // is no cross-compilation and no hard-coded host name anywhere in here. That
7 + // matters more than usual here: the two legs are different architectures on
8 + // different machines, and a recipe that named hosts could install the aarch64
9 + // binary on the Hetzner box.
10 + //
11 + // Unlike pom's recipe this one ends at `deploy`, not `collect`. wam installs
12 + // itself rather than handing off to Sando, because a bad wam deploy costs an
13 + // internal ticket list rather than anyone's platform or the monitoring that
14 + // watches it. The reasoning is written out in bento.toml next to the
15 + // `[[deploy]]` tables; the short form is that the ladder is a real cost and
16 + // belongs where a bad deploy is expensive.
17 +
18 + let h = build_host();
19 + let v = version();
20 + let r = repo();
21 +
22 + step("checkout");
23 + // Pin to the release tag, not whatever main is at pull time. The daemon runs
24 + // the same pin as a cross-host barrier before either target builds, so both
25 + // arches come from one commit rather than from two machines' idea of main --
26 + // which for a mesh that syncs between the two nodes is the difference between
27 + // one protocol and two.
28 + let sha = checkout_sha(h);
29 + log("pinned " + h + " to wam-v" + v + " @ " + sha);
30 +
31 + step("prebuild");
32 + sh_ok(h, "cd " + r + " && cargo clippy --all-targets " + feature_flags() + " -- -D warnings");
33 + sh_ok(h, "cd " + r + " && cargo test " + feature_flags());
34 +
35 + step("build");
36 + sh_ok(h, "cd " + r + " && cargo build --release " + feature_flags());
37 + let binary = resolve_artifact(h, r + "/target/release/wam");
38 +
39 + step("verify");
40 + // The build host must not produce a binary the service host's glibc is too old
41 + // to exec. Bento can make this check here, unlike in pom's handed-off recipe,
42 + // precisely because wam keeps its `[[deploy]]` entry: the check needs to know
43 + // which machine runs the binary, and that entry is where that is written.
44 + log(glibc_check(binary));
45 +
46 + // The version that is about to ship is the version in the tag. A binary
47 + // reporting something else means the checkout and the release disagree.
48 + sh_ok(h, binary + " --version | grep -qw " + v);
49 +
50 + step("deploy");
51 + // Stages under /var/tmp/bento-deploy/wam/ and calls the privileged installer,
52 + // which keeps the outgoing binary as <dst>.prev, installs atomically, and
53 + // restarts the unit. That .prev file is wam's whole rollback story, and it is
54 + // proportionate: putting it back and restarting is one command.
55 + log(deploy(binary));
56 +
57 + step("health");
58 + // What "healthy" means is the service's business, so the recipe asserts it
59 + // itself against the host it just restarted.
60 + //
61 + // The assertion is 401, not 200, and that is the point of this release. The
62 + // binary being replaced is an April build that predates the require_auth
63 + // middleware: it answered /tickets with 200 and the full ticket body to an
64 + // unauthenticated caller. A 200 here would mean the new binary did not take.
65 + // 401 proves both that wam came back up and that it is refusing anonymous
66 + // reads, which is the one thing this deploy exists to change.
67 + sh_ok(deploy_host(), "test -x " + install_path());
68 + sh_ok(deploy_host(), "code=$(curl -s -o /dev/null -w '%{http_code}' " + health_url() + "); test \"$code\" = 401 || { echo \"wam answered $code, expected 401 -- auth is not being enforced\"; exit 1; }");
69 +
70 + log("wam " + v + " (" + target() + ") built on " + h + " and live on " + deploy_host());