Skip to main content

max / makenotwork

Put the production sign-off on the tier that actually guards production A tier's gates guard promotion *out* of it. manual_confirm sat only on tier b, where it guards b -> c, and c has provisioned = false -- so it gated nothing. The ship to prod-1 was cleared by tier a's gates alone: node_health + burn_in. And hotfix: true skips burn_in, so a hotfix reached production behind a single gate. That one gate was weaker than it reads, too: health_url was unset on both nodes, so node_health degraded to `systemctl is-active`, which a crash-looping binary satisfies between restarts -- the exact failure mode of the 0.10.14 CDN_BASE_URL crash-loop on prod-1. So: manual_confirm moves onto tier a, and both nodes get a real readiness probe (testnot-1 binds 127.0.0.1:8080, prod-1 binds 0.0.0.0:3000; both verified serving 200 today). manual_confirm stays on b for the day prod-2 is provisioned. hotfix does not skip manual_confirm, only burn_in, so every prod ship now needs an explicit POST /confirm/a. That is the point of the change. Two tests lock the invariant in structurally rather than pinning tier names, so re-provisioning cannot strand the sign-off again: the predecessor of the last provisioned tier must require manual_confirm, and every serving node must have a health_url. Both fail against the old config. The /deploy skill had `/confirm/b` AFTER `/promote/b` -- satisfying the gate that guards nothing, after the irreversible step. It now runs `/confirm/a` before the promote, and notes that the confirmation must be fresher than the version's landing on a.
Author: Max Johnson <me@maxj.phd> · 2026-07-21 21:48 UTC
Commit: 975f92b319b645305da1a284b77ce9c46f6cc5d2
Parent: 3efc1fe
3 files changed, +84 insertions, -4 deletions
M sando/README.md +7 -1
@@ -118,6 +118,11 @@ Promote-time gates run against a tier's deployed nodes or its operator.
118 118 | `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. |
119 119 | `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. |
120 120
121 + A tier's gates guard promotion **out** of it. So the gate list that stands
122 + between staging and production is tier `a`'s, not tier `b`'s — sign-off for a
123 + prod ship is `POST /confirm/a`, run after the version lands on `a` (the
124 + confirmation must be fresher than that landing) and before `POST /promote/b`.
125 +
121 126 ### What `cargo_test` runs
122 127
123 128 The gate used to be hardcoded to `worktree/server`, so every other crate in the
@@ -229,7 +234,8 @@ Action results show up in the events log a moment later (the actions themselves
229 234 `POST /promote/{tier}` accepts:
230 235
231 236 - `hotfix: true` — skips the `burn_in` gate on the predecessor tier only. All
232 - other gates still apply.
237 + other gates still apply, `manual_confirm` included: a hotfix still needs an
238 + operator sign-off before it reaches production.
233 239 - `reset_burn_in: true` (default `false`) — additionally nulls
234 240 `tier_state.burn_in_started_at` on the source tier, restarting the clock
235 241 for whatever else is still burning in there. Use this only when the hotfix
M sando/daemon/src/topology.rs +54
@@ -399,4 +399,58 @@ service_name = "mnw-cli.service"
399 399 let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../sando.toml");
400 400 Topology::load(&path).expect("shipped sando.toml must validate");
401 401 }
402 +
403 + fn shipped() -> Topology {
404 + let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../sando.toml");
405 + Topology::load(&path).expect("shipped sando.toml must validate")
406 + }
407 +
408 + #[test]
409 + fn shipping_to_the_last_provisioned_tier_needs_an_operator_signoff() {
410 + // A tier's gates guard promotion *out* of it, which is the subtlety that
411 + // made manual_confirm inert: it sat on tier b, guarding b -> c, and c is
412 + // not provisioned. So the ship to production was cleared by node_health
413 + // + burn_in alone — and `hotfix: true` skips burn_in.
414 + //
415 + // The gate that matters therefore belongs on the PREDECESSOR of the last
416 + // provisioned tier. Asserted structurally so re-provisioning tiers cannot
417 + // silently strand the sign-off again.
418 + let topo = shipped();
419 + let last = topo
420 + .tiers
421 + .iter()
422 + .rposition(|t| t.provisioned)
423 + .expect("some tier must be provisioned");
424 + assert!(last > 0, "the production tier cannot be the first tier");
425 + let guard = &topo.tiers[last - 1];
426 + assert!(
427 + guard.gates.iter().any(|g| matches!(g, Gate::ManualConfirm)),
428 + "tier {} guards promotion into the last provisioned tier ({}), so it must require an operator sign-off; its gates are {:?}",
429 + guard.name,
430 + topo.tiers[last].name,
431 + guard
432 + .gates
433 + .iter()
434 + .map(|g| g.kind().as_str())
435 + .collect::<Vec<_>>(),
436 + );
437 + }
438 +
439 + #[test]
440 + fn every_serving_node_has_a_readiness_probe() {
441 + // Without health_url, node_health degrades to `systemctl is-active`,
442 + // which a crash-looping binary satisfies between restarts — exactly what
443 + // the 0.10.14 CDN_BASE_URL crash-loop did on prod-1.
444 + let topo = shipped();
445 + for tier in topo.tiers.iter().filter(|t| t.provisioned) {
446 + for node in &tier.nodes {
447 + assert!(
448 + node.health_url.is_some(),
449 + "node {} on tier {} has no health_url, so node_health proves only that systemd thinks the unit is running",
450 + node.name,
451 + tier.name,
452 + );
453 + }
454 + }
455 + }
402 456 }
M sando/sando.toml +23 -3
@@ -76,9 +76,20 @@ canary = "sequential"
76 76 # node_health (post-deploy) probes the deployed node over its executor — the
77 77 # gate boot_smoke used to stand in for, but boot_smoke runs on the build host and
78 78 # proves nothing about testnot-1. boot_smoke stays a host build-time gate only.
79 + #
80 + # A tier's gates guard promotion *out* of it, so THIS list is what stands between
81 + # testnot and production. manual_confirm lives here for that reason. It used to
82 + # sit only on tier b, where it guards b -> c — and c is not provisioned, so it
83 + # gated nothing: a prod ship was cleared by node_health + burn_in alone, and
84 + # hotfix: true skips burn_in, leaving one gate on a production deploy.
85 + #
86 + # hotfix does NOT skip manual_confirm (only burn_in), so every ship to prod-1
87 + # now needs an explicit POST /confirm/a naming the version. That is the point:
88 + # there was no human sign-off anywhere on the path to production.
79 89 gates = [
80 90 { kind = "node_health" },
81 91 { kind = "burn_in", hours = 48 },
92 + { kind = "manual_confirm" },
82 93 ]
83 94 [[tier.node]]
84 95 name = "testnot-1"
@@ -91,15 +102,20 @@ service_name = "makenotwork.service"
91 102 # fails the promote with the running service intact. The node runs 0.10.14, which
92 103 # supports the mode; validated end-to-end 2026-07-10.
93 104 config_check_env_file = "/etc/mnw/makenotwork.env"
94 - # Set health_url to add an HTTP readiness probe on top of `systemctl is-active`,
95 - # e.g. health_url = "http://127.0.0.1:8080/health" (the local port the service
96 - # binds). Unset = unit-active check only.
105 + # Readiness probe on top of `systemctl is-active`. Without this, node_health
106 + # proves only that systemd thinks the unit is running — which a crash-looping
107 + # binary can satisfy between restarts. The node binds HOST=127.0.0.1 PORT=8080
108 + # (/etc/mnw/makenotwork.env); verified serving 200 on 2026-07-21.
109 + health_url = "http://127.0.0.1:8080/health"
97 110
98 111 # ---- B: prod-1 ----
99 112 [[tier]]
100 113 name = "b"
101 114 provisioned = true
102 115 canary = "sequential"
116 + # Guards b -> c. Kept for the day prod-2 is provisioned; it gates nothing today
117 + # (c has provisioned = false). The sign-off that matters for shipping to THIS
118 + # tier lives on tier a.
103 119 gates = [
104 120 { kind = "node_health" },
105 121 { kind = "manual_confirm" },
@@ -116,6 +132,10 @@ service_name = "makenotwork.service"
116 132 # 0.10.14 deploy crash-looped on a missing CDN_BASE_URL — the exact miss this
117 133 # gate closes. First real exercise is the next prod promote.
118 134 config_check_env_file = "/etc/mnw/makenotwork.env"
135 + # Readiness probe (see testnot-1). The unit binds HOST=0.0.0.0 PORT=3000, so the
136 + # probe dials it on loopback; verified serving 200 on 2026-07-21. This is the
137 + # node whose crash-loop `systemctl is-active` alone would have missed.
138 + health_url = "http://127.0.0.1:3000/health"
119 139 # Companion: install the mnw-cli built in this same promote (see [[companion]] in
120 140 # sando-daemon.toml) after the server is up, and restart its unit. Closes the
121 141 # drift that broke git hosting during the 0.10.14 deploy — mnw-cli now ships from