Skip to main content

max / makenotwork

sando: key gate evidence on the build, not the version string Three defects found by /deepdeploy on 2026-08-19, all of them the same mistake in different places: a version is a label, and a build is the identity. Migration 008 said so and moved the promote path over; the reporting path and the post-deploy gates were never followed through. Gate rows and gate logs are addressed per build. /state keys on the tier's current_build_id, /runs/{id} on the run's own id, and gate logs land under <build_id>/ instead of <version>/. A rebuild at an unchanged version is the normal way to retry a red build, and it used to make the two runs' rows interleave: run 61 of mnw-server 0.11.20 settled with "code_smoke: migrate+seed run failed" while the same response listed code_smoke passed, and /runs/62's cargo_deny.log held runs 60 and 61's output. failure_summary now reads the failing run's own rows. manual_confirm records the build it confirms, so it can be scoped too. node_health and page_smoke are evidence about a tier, not about the bytes leaving it, so they are checked once against the tier's own build. Checked per shipped build they made a cross-architecture promote structurally impossible: pom 0.4.3 astra -> hetzner ships the x86_64 bundle, astra has only ever run the aarch64 one, and astra can never acquire a node_health row for a build it cannot execute. The artifact gates stay per build, which is what they are for. In the same pass, a promote records the build the target tier actually received on its deploys rows and its tier_state, rather than the source tier's. /self-update fetches the deploy branch from the canonical remote before the ancestry check, and answers a non-deploy-branch sha synchronously instead of exiting 4 in the updater's journal. The bare repo used to advance only when a server build fetched, which chained the controller's currency to the server's release cadence and blocked a controller fix outright while the server was red. sandod is still the only writer of that repo, so the provenance seal is unchanged.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-20 21:14 UTC
Signed with PGP, not checked
Commit: 052fd3809891c653484898ac22293b538eea0a8a
Parent: 54178b4
11 files changed, +621 insertions, -184 deletions
M sando/Cargo.lock +15 -15
@@ -1227,7 +1227,7 @@
1227 1227
1228 1228 [[package]]
1229 1229 name = "sando-daemon"
1230 - version = "0.2.8"
1230 + version = "0.2.9"
1231 1231 dependencies = [
1232 1232 "anyhow",
1233 1233 "async-trait",
@@ -2289,29 +2289,33 @@
2289 2289 name = "tagtree"
2290 2290 version = "0.4.0"
2291 2291
2292 + [[patch.unused]]
2293 + name = "quasi-type"
2294 + version = "0.1.0"
2295 +
2292 2296 [[patch.unused]]
2293 2297 name = "quasi-axum"
2294 - version = "0.36.0"
2298 + version = "0.44.0"
2295 2299
2296 2300 [[patch.unused]]
2297 2301 name = "quasi-basics"
2298 - version = "0.36.0"
2302 + version = "0.44.0"
2299 2303
2300 2304 [[patch.unused]]
2301 2305 name = "quasi-http"
2302 - version = "0.36.0"
2306 + version = "0.44.0"
2303 2307
2304 2308 [[patch.unused]]
2305 2309 name = "quasi-immediate"
2306 - version = "0.36.0"
2310 + version = "0.44.0"
2307 2311
2308 2312 [[patch.unused]]
2309 2313 name = "quasi-notifs"
2310 - version = "0.36.0"
2314 + version = "0.44.0"
2311 2315
2312 2316 [[patch.unused]]
2313 2317 name = "quasi-router"
2314 - version = "0.36.0"
2318 + version = "0.44.0"
2315 2319
2316 2320 [[patch.unused]]
2317 2321 name = "quasi-store"
@@ -2319,19 +2323,15 @@
2319 2323
2320 2324 [[patch.unused]]
2321 2325 name = "quasi-tauri"
2322 - version = "0.36.0"
2326 + version = "0.44.0"
2323 2327
2324 2328 [[patch.unused]]
2325 2329 name = "quasi-webview"
2326 - version = "0.36.0"
2327 -
2328 - [[patch.unused]]
2329 - name = "quasi-type"
2330 - version = "0.1.0"
2330 + version = "0.44.0"
2331 2331
2332 2332 [[patch.unused]]
2333 2333 name = "makeover-build"
2334 - version = "0.46.0"
2334 + version = "0.47.0"
2335 2335
2336 2336 [[patch.unused]]
2337 2337 name = "makeover-immediate"
@@ -2351,4 +2351,4 @@
2351 2351
2352 2352 [[patch.unused]]
2353 2353 name = "makeover-webview"
2354 - version = "0.51.0"
2354 + version = "0.52.0"
M sando/daemon/Cargo.toml +1 -1
@@ -1,6 +1,6 @@
1 1 [package]
2 2 name = "sando-daemon"
3 - version = "0.2.8"
3 + version = "0.2.9"
4 4 edition = "2024"
5 5 license = "MIT"
6 6
M sando/deploy/sando-self-update.sh +12 -7
@@ -21,13 +21,18 @@
21 21 # SANDO_DAEMON_CONFIG /etc/sando/sando-daemon.toml config the --check-config self-test loads
22 22 #
23 23 # Source: build from the LOCAL bare repo sandod already maintains
24 - # (/srv/sando/mnw.git), not a remote fetch. The self-update sha is one that was
25 - # already deployed as a server release, so `/rebuild` has force-updated
26 - # refs/heads/main in the bare repo to include it — the sha and a current, trusted
27 - # main are both present locally. Building from a remote (git@ssh.makenot.work)
28 - # gave the `sando` user no git creds and broke the moment git hosting was down —
29 - # it blocked the self-update during the 2026-07-09 deploy (postmortem #7). The
30 - # local bare repo has no such external dependency.
24 + # (/srv/sando/mnw.git), not a remote fetch. Building from a remote
25 + # (git@ssh.makenot.work) gave the `sando` user no git creds and broke the moment
26 + # git hosting was down — it blocked the self-update during the 2026-07-09 deploy
27 + # (postmortem #7). The local bare repo has no such external dependency.
28 + #
29 + # sandod fetches the deploy branch from the canonical remote into that bare repo
30 + # at the top of its /self-update handler, so a sha pushed minutes ago is present
31 + # here. It used to be reachable only if a *server* build had fetched since, which
32 + # chained the controller's currency to the server's release cadence: shipping a
33 + # controller fix meant cutting a server release first, and was impossible at all
34 + # while the server was red. sandod remains the only writer of that repo, so the
35 + # provenance seal below is unchanged.
31 36 #
32 37 # Provenance: sandod (bearer-gated) only *triggers* this unit with a hex sha; it
33 38 # does not prove the sha is a commit anyone intended to deploy. Without a check,
M sando/daemon/src/build.rs +1 -1
@@ -770,7 +770,7 @@
770 770 } else {
771 771 // Pull the first red gate's typed summary into the run so the API
772 772 // answers "which gate, and why" — not just "failed".
773 - let summary = crate::runs::first_failed_gate_summary(&pool, &cfg.id, &published.version)
773 + let summary = crate::runs::first_failed_gate_summary(&pool, &cfg.id, run_id)
774 774 .await
775 775 .unwrap_or_else(|| "host pipeline red".to_string());
776 776 if let Err(e) = crate::runs::mark_failed(&pool, run_id, &summary).await {
M sando/daemon/src/config.rs +3 -2
@@ -146,8 +146,9 @@
146 146 /// `["makenotwork", "mnw-admin"]`.
147 147 #[serde(default = "default_bin_names")]
148 148 pub bin_names: Vec<String>,
149 - /// Root for per-gate run logs (`<logs_root>/<version>/<gate>.log`).
150 - /// Served via `GET /logs/{version}/{gate}`. Defaults to `/srv/sando/logs`.
149 + /// Root for per-gate run logs (`<logs_root>/<build_runs.id>/<gate>.log`).
150 + /// Served via `GET /logs/{run}/{gate}`, where `run` is the `build_runs.id` the
151 + /// gate ran for. Defaults to `/srv/sando/logs`.
151 152 #[serde(default = "default_logs_root")]
152 153 pub logs_root: PathBuf,
153 154 /// Shared cargo target dir. When set, every `cargo build`/`cargo test` the
M sando/daemon/src/gates.rs +49 -25
@@ -81,6 +81,37 @@
81 81 }
82 82
83 83 impl GateCtx {
84 + /// The `logs_root` sub-directory this run's gate logs land in.
85 + ///
86 + /// The build id, so two runs of one version keep two sets of logs. Before
87 + /// 2026-08-20 this was the version, which meant a rebuild appended to the
88 + /// previous attempt's file and every run pointed at the same mixed log.
89 + ///
90 + /// Falls back to the version when there is no build identity — a
91 + /// pre-migration-008 run, or a gate unit test. That is also what keeps logs
92 + /// written under the old scheme reachable: their rows still record the
93 + /// version path, and nothing rewrites them.
94 + pub fn log_scope(&self) -> String {
95 + self.build_id
96 + .map_or_else(|| self.version.to_string(), |id| id.to_string())
97 + }
98 +
99 + /// This run's log pointer for `gate`. Always paired with
100 + /// [`Self::log_path`], which resolves the same ref to an absolute path.
101 + pub fn log_ref(&self, gate: GateKind) -> LogRef {
102 + LogRef::new(&self.log_scope(), gate)
103 + }
104 +
105 + /// Where `gate`'s log is written on this host: `logs_root` joined to
106 + /// [`Self::log_ref`]. The two are derived from one scope so a row's
107 + /// `log_ref` can never name a file the gate did not write.
108 + pub fn log_path(&self, gate: GateKind) -> PathBuf {
109 + self.cfg
110 + .logs_root
111 + .join(self.log_scope())
112 + .join(format!("{}.log", gate.as_str()))
113 + }
114 +
84 115 /// Absolute directory a `test_target` runs in: under the worktree, or under
85 116 /// the named aux repo's checkout.
86 117 ///
@@ -203,7 +234,7 @@
203 234 gate: GateKind::MigrationDryRun,
204 235 after_s: ctx.cfg.gate_timeout_secs as u32,
205 236 })
206 - .with_log_ref(LogRef::new(&ctx.version, GateKind::MigrationDryRun))),
237 + .with_log_ref(ctx.log_ref(GateKind::MigrationDryRun))),
207 238 }
208 239 }
209 240 // code_smoke boots the real binary (migrate-from-scratch + seed + serve),
@@ -219,7 +250,7 @@
219 250 gate: GateKind::CodeSmoke,
220 251 after_s: ctx.cfg.gate_timeout_secs as u32,
221 252 })
222 - .with_log_ref(LogRef::new(&ctx.version, GateKind::CodeSmoke))),
253 + .with_log_ref(ctx.log_ref(GateKind::CodeSmoke))),
223 254 }
224 255 }
225 256 Gate::BootSmoke => boot_smoke(ctx, run_id).await,
@@ -305,8 +336,8 @@
305 336 /// operator's answer. Its name is carried in the failure so the summary points
306 337 /// at the crate, not just the test.
307 338 async fn cargo_test(ctx: &GateCtx, run_id: GateRunId) -> Result<GateOutcome> {
308 - let log_path = gate_log_path(ctx, GateKind::CargoTest);
309 - let log_ref = LogRef::new(&ctx.version, GateKind::CargoTest);
339 + let log_path = ctx.log_path(GateKind::CargoTest);
340 + let log_ref = ctx.log_ref(GateKind::CargoTest);
310 341
311 342 // Best-effort: drop our own role's stale `mnw_test_*` databases (the
312 343 // template + any per-test clones orphaned by a previously-killed run)
@@ -606,8 +637,8 @@
606 637 kind: GateKind,
607 638 args_for: impl Fn(&crate::config::TestTarget, &std::path::Path) -> Option<Vec<String>>,
608 639 ) -> Result<GateOutcome> {
609 - let log_path = gate_log_path(ctx, kind);
610 - let log_ref = LogRef::new(&ctx.version, kind);
640 + let log_path = ctx.log_path(kind);
641 + let log_ref = ctx.log_ref(kind);
611 642 let started = std::time::Instant::now();
612 643 let deadline = started + std::time::Duration::from_secs(ctx.cfg.gate_timeout_secs);
613 644 let mut ran = 0usize;
@@ -763,8 +794,8 @@
763 794 all_features: false,
764 795 scratch_db: true,
765 796 };
766 - let log_path = gate_log_path(ctx, GateKind::HardeningTest);
767 - let log_ref = LogRef::new(&ctx.version, GateKind::HardeningTest);
797 + let log_path = ctx.log_path(GateKind::HardeningTest);
798 + let log_ref = ctx.log_ref(GateKind::HardeningTest);
768 799
769 800 if let Some(scratch_url) = ctx.cfg.scratch_db_url.as_deref() {
770 801 clean_stale_test_dbs(scratch_url).await;
@@ -939,7 +970,7 @@
939 970 let log = GateLog::open(ctx, run_id, GateKind::MigrationDryRun).await;
940 971 let outcome = migration_dry_run_inner(ctx, &log).await;
941 972 log.close().await;
942 - outcome.map(|o| o.with_log_ref(LogRef::new(&ctx.version, GateKind::MigrationDryRun)))
973 + outcome.map(|o| o.with_log_ref(ctx.log_ref(GateKind::MigrationDryRun)))
943 974 }
944 975
945 976 /// The staged interior of [`migration_dry_run`], writing every step through the
@@ -1457,7 +1488,7 @@
1457 1488 let log = GateLog::open(ctx, run_id, GateKind::CodeSmoke).await;
1458 1489 let outcome = code_smoke_inner(ctx, &log).await;
1459 1490 log.close().await;
1460 - outcome.map(|o| o.with_log_ref(LogRef::new(&ctx.version, GateKind::CodeSmoke)))
1491 + outcome.map(|o| o.with_log_ref(ctx.log_ref(GateKind::CodeSmoke)))
1461 1492 }
1462 1493
1463 1494 /// The staged interior of [`code_smoke`], writing every step through the gate's
@@ -2002,8 +2033,8 @@
2002 2033 if let Some(scratch_url) = ctx.cfg.scratch_db_url.as_deref() {
2003 2034 cmd.env("DATABASE_URL", scratch_url);
2004 2035 }
2005 - let log_path = gate_log_path(ctx, GateKind::BootSmoke);
2006 - let log_ref = LogRef::new(&ctx.version, GateKind::BootSmoke);
2036 + let log_path = ctx.log_path(GateKind::BootSmoke);
2037 + let log_ref = ctx.log_ref(GateKind::BootSmoke);
2007 2038 let mut child = match cmd.spawn() {
2008 2039 Ok(c) => c,
2009 2040 Err(e) => {
@@ -2190,7 +2221,7 @@
2190 2221 gate: GateKind::PageSmoke,
2191 2222 after_s: ctx.cfg.gate_timeout_secs as u32,
2192 2223 })
2193 - .with_log_ref(LogRef::new(&ctx.version, GateKind::PageSmoke)));
2224 + .with_log_ref(ctx.log_ref(GateKind::PageSmoke)));
2194 2225 }
2195 2226 };
2196 2227
@@ -2199,12 +2230,12 @@
2199 2230 String::from_utf8_lossy(&out.stdout),
2200 2231 String::from_utf8_lossy(&out.stderr)
2201 2232 );
2202 - append_to_log(&gate_log_path(ctx, GateKind::PageSmoke), log.as_bytes()).await;
2233 + append_to_log(&ctx.log_path(GateKind::PageSmoke), log.as_bytes()).await;
2203 2234
2204 2235 if out.status.success() {
2205 2236 return Ok(
2206 2237 GateOutcome::passed(PassNote::PagesClean { base: base.into() })
2207 - .with_log_ref(LogRef::new(&ctx.version, GateKind::PageSmoke)),
2238 + .with_log_ref(ctx.log_ref(GateKind::PageSmoke)),
2208 2239 );
2209 2240 }
2210 2241
@@ -2222,7 +2253,7 @@
2222 2253 base: base.into(),
2223 2254 detail: first.to_string(),
2224 2255 })
2225 - .with_log_ref(LogRef::new(&ctx.version, GateKind::PageSmoke)))
2256 + .with_log_ref(ctx.log_ref(GateKind::PageSmoke)))
2226 2257 }
2227 2258
2228 2259 async fn node_health(ctx: &GateCtx) -> Result<GateOutcome> {
@@ -2359,7 +2390,7 @@
2359 2390 async fn open(ctx: &GateCtx, run_id: GateRunId, gate: GateKind) -> Self {
2360 2391 Self::new(
2361 2392 LiveLog::open(
2362 - gate_log_path(ctx, gate),
2393 + ctx.log_path(gate),
2363 2394 gate_chunk_cb(ctx.events.clone(), run_id),
2364 2395 )
2365 2396 .await,
@@ -2433,13 +2464,6 @@
2433 2464 }
2434 2465 }
2435 2466
2436 - fn gate_log_path(ctx: &GateCtx, gate: GateKind) -> PathBuf {
2437 - ctx.cfg
2438 - .logs_root
2439 - .join(ctx.version.to_string())
2440 - .join(format!("{}.log", gate.as_str()))
2441 - }
2442 -
2443 2467 /// Live check: has `tier`'s burn-in window of `hours` elapsed since its clock
2444 2468 /// (`tier_state.burn_in_started_at`, started by a promote onto the tier)? Used
2445 2469 /// by the promote-time gate check (`unsatisfied_gates`) so a stale `blocked`
@@ -2958,7 +2982,7 @@
2958 2982 /// Close `log` (flushing it) and read back what it wrote on disk.
2959 2983 async fn read_gate_log(ctx: &GateCtx, log: GateLog) -> String {
2960 2984 log.close().await;
2961 - tokio::fs::read_to_string(gate_log_path(ctx, GateKind::CodeSmoke))
2985 + tokio::fs::read_to_string(ctx.log_path(GateKind::CodeSmoke))
2962 2986 .await
2963 2987 .expect("the gate log must exist on disk")
2964 2988 }
M sando/daemon/src/git.rs +64
@@ -92,6 +92,27 @@
92 92 Ok(())
93 93 }
94 94
95 + /// True iff `sha` is reachable from `refs/heads/<branch>` in the bare repo.
96 + ///
97 + /// The same provenance seal `deploy/sando-self-update.sh` enforces before it
98 + /// builds, evaluated here so `/self-update` can refuse a non-deploy-branch sha
99 + /// synchronously instead of returning `accepted: true` and failing with exit 4
100 + /// in the updater unit's journal, where nobody is watching.
101 + ///
102 + /// Fail-closed: `git merge-base --is-ancestor` exits 1 for a non-ancestor and
103 + /// >1 for an unresolvable ref, and both read as "not an ancestor" here.
104 + pub async fn is_ancestor(bare: &Path, sha: &str, branch: &str) -> Result<bool> {
105 + let out = Command::new("git")
106 + .arg("--git-dir")
107 + .arg(bare)
108 + .args(["merge-base", "--is-ancestor", sha])
109 + .arg(format!("refs/heads/{branch}"))
110 + .output()
111 + .await
112 + .context("spawning git merge-base --is-ancestor")?;
113 + Ok(out.status.success())
114 + }
115 +
95 116 /// True if `sha` resolves to a commit object in the bare repo. Used to give a
96 117 /// clear "push first" error instead of a cryptic `git worktree add` failure.
97 118 pub async fn sha_present(bare: &Path, sha: &str) -> Result<bool> {
@@ -302,6 +323,49 @@
302 323 assert!(!without.join("hooks/post-receive").exists());
303 324 }
304 325
326 + #[tokio::test]
327 + async fn is_ancestor_seals_the_deploy_branch() {
328 + // The provenance seal `/self-update` checks before it triggers the
329 + // privileged updater: a sha on the deploy branch passes, anything else
330 + // is refused fail-closed. `two_commit_repo` puts both commits on `main`.
331 + let (tmp, gitdir, sha1, sha2) = two_commit_repo().await;
332 + assert!(is_ancestor(&gitdir, &sha1, "main").await.unwrap());
333 + assert!(is_ancestor(&gitdir, &sha2, "main").await.unwrap());
334 +
335 + // A commit that exists but is not on the deploy branch.
336 + let repo = tmp.path().join("repo");
337 + assert!(
338 + git(&repo, &["checkout", "-q", "-b", "side", &sha1])
339 + .await
340 + .status
341 + .success()
342 + );
343 + tokio::fs::write(repo.join("c.txt"), b"three")
344 + .await
345 + .unwrap();
346 + assert!(git(&repo, &["add", "."]).await.status.success());
347 + assert!(
348 + git(&repo, &["commit", "-q", "-m", "three"])
349 + .await
350 + .status
351 + .success()
352 + );
353 + let side = rev_parse(&repo, "HEAD").await;
354 + assert!(
355 + !is_ancestor(&gitdir, &side, "main").await.unwrap(),
356 + "a feature-branch tip is not deployable",
357 + );
358 +
359 + // An unresolvable ref is "not an ancestor", never an error that a caller
360 + // could mistake for a pass.
361 + assert!(
362 + !is_ancestor(&gitdir, "0123456789abcdef0123456789abcdef01234567", "main")
363 + .await
364 + .unwrap()
365 + );
366 + assert!(!is_ancestor(&gitdir, &sha1, "no-such-branch").await.unwrap());
367 + }
368 +
305 369 #[tokio::test]
306 370 async fn checkout_worktree_creates_at_sha() {
307 371 let (tmp, gitdir, sha1, _sha2) = two_commit_repo().await;
M sando/daemon/src/outcome.rs +28 -7
@@ -504,16 +504,29 @@
504 504 }
505 505
506 506 /// Pointer to the on-disk gate log: a path relative to `cfg.logs_root`
507 - /// of the form `<version>/<gate_kind>.log`. Stored in `gate_runs.log_ref`
508 - /// and surfaced in `/state` so the TUI/operator can request the full
509 - /// tail via `GET /logs/<version>/<gate>` only when needed.
507 + /// of the form `<scope>/<gate_kind>.log`. Stored in `gate_runs.log_ref` and
508 + /// surfaced in `/state` so the TUI/operator can request the full tail via
509 + /// `GET /logs/<scope>/<gate>` only when needed.
510 + ///
511 + /// **`scope` is the `build_runs.id` the gate ran for**, not the version. It was
512 + /// the version until 2026-08-20, and that made every run of a version write the
513 + /// same file: a rebuild at an unchanged version — which is the normal way to
514 + /// retry a red build — appended its output to the previous attempt's log, and
515 + /// every one of those runs' `log_ref`s pointed at the shared file. Reading
516 + /// `/runs/62`'s `cargo_deny.log` returned runs 60 and 61's output, naming a
517 + /// dependency failure that run 62's worktree had already fixed.
518 + ///
519 + /// [`GateCtx::log_ref`](crate::gates::GateCtx::log_ref) is what builds these;
520 + /// it falls back to the version string for a run with no build identity
521 + /// (pre-migration-008 rows and gate unit tests), which is also why logs written
522 + /// under the old scheme stay readable at the path their row already records.
510 523 #[derive(Debug, Clone, Serialize, Deserialize)]
511 524 #[serde(transparent)]
512 525 pub struct LogRef(pub String);
513 526
514 527 impl LogRef {
515 - pub fn new(version: &Version, gate: GateKind) -> Self {
516 - Self(format!("{}/{}.log", version, gate.as_str()))
528 + pub fn new(scope: &str, gate: GateKind) -> Self {
529 + Self(format!("{}/{}.log", scope, gate.as_str()))
517 530 }
518 531 pub fn as_str(&self) -> &str {
519 532 &self.0
@@ -575,9 +588,17 @@
575 588
576 589 #[test]
577 590 fn log_ref_construction_matches_disk_layout() {
591 + // A build-scoped ref: two runs of one version get two files.
592 + assert_eq!(
593 + LogRef::new("62", GateKind::CargoTest).as_str(),
594 + "62/cargo_test.log"
595 + );
596 + // The pre-identity fallback, still addressable at the path old rows record.
578 597 let v: Version = "0.9.6".parse().unwrap();
579 - let lr = LogRef::new(&v, GateKind::CargoTest);
580 - assert_eq!(lr.as_str(), "0.9.6/cargo_test.log");
598 + assert_eq!(
599 + LogRef::new(&v.to_string(), GateKind::CargoTest).as_str(),
600 + "0.9.6/cargo_test.log"
601 + );
581 602 }
582 603
583 604 #[test]
M sando/daemon/src/runs.rs +51 -33
@@ -292,32 +292,41 @@
292 292 // belongs to. Asking the caller would let a lookup for one product's run
293 293 // return another's gates.
294 294 let app: String = row.get("app");
295 - // Gates are keyed by (app, tier, version); a build run drives the `host` tier.
296 - // Latest row per gate_kind, matching `/state`'s per-tier query shape.
297 - let gates: Vec<RunGateView> = if let Some(ver) = version.as_deref() {
298 - sqlx::query(
299 - "SELECT gate_kind, status, log_ref
300 - FROM gate_runs g
301 - WHERE app = ?1 AND tier = 'host' AND version = ?2
302 - AND id = (SELECT MAX(id) FROM gate_runs
303 - WHERE app = ?1 AND tier = 'host' AND version = ?2
304 - AND gate_kind = g.gate_kind)
305 - ORDER BY gate_kind",
306 - )
307 - .bind(&app)
308 - .bind(ver)
309 - .fetch_all(pool)
310 - .await?
311 - .into_iter()
312 - .map(|gr| RunGateView {
313 - kind: gr.get("gate_kind"),
314 - status: gr.get("status"),
315 - log_ref: gr.get("log_ref"),
316 - })
317 - .collect()
318 - } else {
319 - Vec::new()
320 - };
295 + // **This run's gates, keyed on this run.** A build run's id IS the build
296 + // identity every gate it ran recorded (`gate_runs.build_id`), so asking for
297 + // them is an exact question with an exact answer.
298 + //
299 + // It was keyed on (tier, version) until 2026-08-20, and a rebuild at an
300 + // unchanged version is the normal way to retry a red build: runs 60, 61 and
301 + // 62 of mnw-server 0.11.20 wrote over each other's rows, and `/runs/62`
302 + // answered with whichever run had touched each gate last. Two reads a minute
303 + // apart, same run, no promote between them, disagreed about whether
304 + // `hardening_test` had passed or not run at all.
305 + //
306 + // A pre-migration-008 run left rows with a NULL `build_id` and so reports no
307 + // gates here. That is the honest answer for a row that never recorded which
308 + // build it vouched for, and the same call the migration made in refusing to
309 + // backfill one.
310 + let gates: Vec<RunGateView> = sqlx::query(
311 + "SELECT gate_kind, status, log_ref
312 + FROM gate_runs g
313 + WHERE app = ?1 AND build_id = ?2
314 + AND id = (SELECT MAX(id) FROM gate_runs
315 + WHERE app = ?1 AND build_id = ?2
316 + AND gate_kind = g.gate_kind)
317 + ORDER BY gate_kind",
318 + )
319 + .bind(&app)
320 + .bind(run_id.0)
321 + .fetch_all(pool)
322 + .await?
323 + .into_iter()
324 + .map(|gr| RunGateView {
325 + kind: gr.get("gate_kind"),
326 + status: gr.get("status"),
327 + log_ref: gr.get("log_ref"),
328 + })
329 + .collect();
321 330
322 331 Ok(Some(RunView {
323 332 run_id: row.get("id"),
@@ -387,22 +396,31 @@
387 396 (end - start.with_timezone(&Utc)).num_seconds().max(0)
388 397 }
389 398
390 - /// The summary of the first failed gate for `version` on the host tier, if
391 - /// any — used by the build pipeline to populate `failure_summary` when
392 - /// `run_all` reports a red pipeline. Reads the typed `outcome_json` so the
393 - /// stored headline matches what the TUI renders.
399 + /// The summary of the first gate `run_id` failed, if any — used by the build
400 + /// pipeline to populate `failure_summary` when `run_all` reports a red
401 + /// pipeline. Reads the typed `outcome_json` so the stored headline matches what
402 + /// the TUI renders.
403 + ///
404 + /// Keyed on the run, not on (tier, version). Version-keyed, this read the first
405 + /// failure any run of that version had ever recorded, so a rebuild inherited its
406 + /// predecessor's headline: run 61 of mnw-server 0.11.20 settled `failed` with
407 + /// `failure_summary: "code_smoke: migrate+seed run failed: exit 1"` while the
408 + /// same response listed `code_smoke passed`. The `/deploy` runbook says to
409 + /// surface `failure_summary` to the operator, so the documented procedure was
410 + /// pointing at a green gate — it cost a wasted repin of seventeen seed digests
411 + /// that turned out to be correct.
394 412 pub async fn first_failed_gate_summary(
395 413 pool: &SqlitePool,
396 414 app: &AppId,
397 - version: &Version,
415 + run_id: RunId,
398 416 ) -> Option<String> {
399 417 let row = sqlx::query(
400 418 "SELECT gate_kind, outcome_json FROM gate_runs
401 - WHERE app = ? AND tier = 'host' AND version = ? AND status = 'failed'
419 + WHERE app = ? AND build_id = ? AND status = 'failed'
402 420 ORDER BY id ASC LIMIT 1",
403 421 )
404 422 .bind(app)
405 - .bind(version.to_string())
423 + .bind(run_id.0)
406 424 .fetch_optional(pool)
407 425 .await
408 426 .ok()
M sando/daemon/src/routes/mod.rs +270 -69
@@ -35,7 +35,7 @@
35 35 .route("/status.json", get(get_status_json))
36 36 .route("/runs/{id}", get(get_run))
37 37 .route("/runs/{id}/wait", get(get_run_wait))
38 - .route("/logs/{version}/{gate}", get(get_gate_log))
38 + .route("/logs/{run}/{gate}", get(get_gate_log))
39 39 .route("/events", get(events_ws))
40 40 .route_layer(axum::middleware::from_fn(move |req, next| {
41 41 require_bearer(token.clone(), req, next)
@@ -208,7 +208,7 @@
208 208 let rows = sqlx::query(
209 209 "SELECT t.name, t.ord, t.provisioned, t.canary,
210 210 ts.current_version, ts.previous_version, ts.burn_in_started_at,
211 - ts.partial_reason
211 + ts.partial_reason, ts.current_build_id
212 212 FROM tiers t
213 213 LEFT JOIN tier_state ts ON ts.app = t.app AND ts.tier = t.name
214 214 WHERE t.app = ?
@@ -222,6 +222,7 @@
222 222 for r in rows {
223 223 let name: String = r.get("name");
224 224 let current_version: Option<String> = r.get("current_version");
225 + let current_build_id: Option<i64> = r.get("current_build_id");
225 226
226 227 let nodes: Vec<String> =
227 228 sqlx::query_scalar("SELECT name FROM nodes WHERE app = ? AND tier = ? ORDER BY name")
@@ -230,15 +231,32 @@
230 231 .fetch_all(&s.pool)
231 232 .await?;
232 233
233 - // Surface gates for current_version when set, otherwise for the most
234 - // recently attempted version on this tier. Without the fallback, a
235 - // tier that has never gone green (MM after a build failure, B before
236 - // first deploy) exposes no gate detail via /state — debugging required
237 - // SSH and direct SQLite access. See sando todo: gate observability.
238 - let gate_version: Option<String> = if current_version.is_some() {
239 - current_version.clone()
240 - } else {
241 - sqlx::query_scalar(
234 + // Which run's gates this tier reports.
235 + //
236 + // The build the tier is actually running, when it has an identity: gate
237 + // rows are evidence about a build, and a version string is not one. This
238 + // was keyed on the version until 2026-08-20, and a rebuild at an
239 + // unchanged version — the normal way to retry a red build — made the
240 + // rows of two runs interleave, each gate showing whichever run wrote it
241 + // last. Statuses appeared to move backwards on a tier nobody had touched
242 + // (`hardening_test passed` then `hardening_test None`, a minute apart),
243 + // because a reader cannot tell a not-yet-run gate from one a sibling run
244 + // has just replaced.
245 + //
246 + // `GateScope::Version` is the fallback, and it covers two live cases: a
247 + // pre-identity tier (NULL `current_build_id`, the same legacy path
248 + // `unsatisfied_gates` takes), and a tier that has never gone green at all
249 + // — MM after a build failure, B before its first deploy — where the most
250 + // recently attempted version is the only handle there is. Without that
251 + // second one, debugging a red tier meant SSH and direct SQLite access.
252 + enum GateScope {
253 + Build(i64),
254 + Version(String),
255 + }
256 + let scope: Option<GateScope> = match (current_build_id, current_version.clone()) {
257 + (Some(bid), _) => Some(GateScope::Build(bid)),
258 + (None, Some(ver)) => Some(GateScope::Version(ver)),
259 + (None, None) => sqlx::query_scalar::<_, String>(
242 260 "SELECT version FROM gate_runs WHERE app = ? AND tier = ?
243 261 ORDER BY id DESC LIMIT 1",
244 262 )
@@ -246,24 +264,47 @@
246 264 .bind(&name)
247 265 .fetch_optional(&s.pool)
248 266 .await?
267 + .map(GateScope::Version),
249 268 };
250 269
251 - let gates: Vec<GateView> = if let Some(ver) = gate_version.as_ref() {
252 - // Most recent gate_runs row per gate_kind for (tier, ver).
253 - sqlx::query(
254 - "SELECT gate_kind, finished_at, status, outcome_json, log_ref
255 - FROM gate_runs g
256 - WHERE app = ?1 AND tier = ?2 AND version = ?3
257 - AND id = (SELECT MAX(id) FROM gate_runs
258 - WHERE app = ?1 AND tier = ?2 AND version = ?3
259 - AND gate_kind = g.gate_kind)
260 - ORDER BY gate_kind",
261 - )
262 - .bind(&s.cfg.id)
263 - .bind(&name)
264 - .bind(ver)
265 - .fetch_all(&s.pool)
266 - .await?
270 + // Most recent row per gate_kind within the scope. Both arms share the
271 + // shape; only the key differs.
272 + let gate_rows = match &scope {
273 + Some(GateScope::Build(bid)) => {
274 + sqlx::query(
275 + "SELECT gate_kind, finished_at, status, outcome_json, log_ref
276 + FROM gate_runs g
277 + WHERE app = ?1 AND tier = ?2 AND build_id = ?3
278 + AND id = (SELECT MAX(id) FROM gate_runs
279 + WHERE app = ?1 AND tier = ?2 AND build_id = ?3
280 + AND gate_kind = g.gate_kind)
281 + ORDER BY gate_kind",
282 + )
283 + .bind(&s.cfg.id)
284 + .bind(&name)
285 + .bind(bid)
286 + .fetch_all(&s.pool)
287 + .await?
288 + }
289 + Some(GateScope::Version(ver)) => {
290 + sqlx::query(
291 + "SELECT gate_kind, finished_at, status, outcome_json, log_ref
292 + FROM gate_runs g
293 + WHERE app = ?1 AND tier = ?2 AND version = ?3
294 + AND id = (SELECT MAX(id) FROM gate_runs
295 + WHERE app = ?1 AND tier = ?2 AND version = ?3
296 + AND gate_kind = g.gate_kind)
297 + ORDER BY gate_kind",
298 + )
299 + .bind(&s.cfg.id)
300 + .bind(&name)
301 + .bind(ver)
302 + .fetch_all(&s.pool)
303 + .await?
304 + }
305 + None => Vec::new(),
306 + };
307 + let gates: Vec<GateView> = gate_rows
267 308 .into_iter()
268 309 .map(|gr| GateView {
269 310 kind: gr.get("gate_kind"),
@@ -274,10 +315,7 @@
274 315 .and_then(|s| serde_json::from_str(&s).ok()),
275 316 log_ref: gr.get("log_ref"),
276 317 })
277 - .collect()
278 - } else {
279 - Vec::new()
280 - };
318 + .collect();
281 319
282 320 tiers.push(TierView {
283 321 name,
@@ -793,6 +831,24 @@
793 831 format!("sando-update@{sha}.service")
794 832 }
795 833
834 + /// The bare repo sandod's own source lives in, and the branch that seals it.
835 + ///
836 + /// sandod is built from `sando/daemon` inside the **default product's** repo, so
837 + /// that product's `[repo]` is the one the self-update path has to keep current.
838 + /// Resolved here rather than from `s.topo` because `/self-update` is mounted
839 + /// under every product (`/apps/pom/self-update` reaches the same daemon), and
840 + /// pom's topology is intake-only — it declares no `[repo]` at all. Reading the
841 + /// per-request product would make the fetch below silently a no-op on every
842 + /// mount but one.
843 + fn self_update_source(s: &AppState) -> Option<(std::path::PathBuf, String, Option<String>)> {
844 + let repo = s.apps.get(&s.default_app)?.topo.repo.as_ref()?;
845 + Some((
846 + std::path::PathBuf::from(&repo.bare_path),
847 + repo.branch.clone(),
848 + repo.upstream.clone(),
849 + ))
850 + }
851 +
796 852 /// Trigger a rebuild + restart of sandod *itself* to `sha`. sandod runs
797 853 /// unprivileged (User=sando, NoNewPrivileges, ProtectSystem=strict) and cannot
798 854 /// write `/usr/local/bin/sandod` or restart its own service — so it only
@@ -801,6 +857,16 @@
801 857 /// `sando/daemon` as the sando user in a dedicated checkout, installs the new
802 858 /// binary, and restarts sandod. Bearer-gated like the other mutators; the new
803 859 /// version shows up in `/state`'s `sandod_version` once the restart lands.
860 + ///
861 + /// **The upstream fetch below is what makes a fresh sha reachable at all.** The
862 + /// updater builds from the local bare repo and refuses any sha that is not an
863 + /// ancestor of its `main` (see `deploy/sando-self-update.sh`, "Provenance"). That
864 + /// bare repo used to advance only when `/rebuild` fetched upstream during a
865 + /// *server* build, which chained the controller's currency to the server's
866 + /// release cadence and meant an urgent controller fix could not ship while the
867 + /// server was red. Fetching here decouples the two: the rule stays exactly as
868 + /// strict, it is simply checked against the canonical remote rather than against
869 + /// a local cache of it that only a server build refreshed.
804 870 async fn self_update(
805 871 State(s): State<AppState>,
806 872 crate::error::TypedBody(body): crate::error::TypedBody<SelfUpdateBody>,
@@ -810,6 +876,40 @@
810 876
811 877 let unit = self_update_unit(&sha);
812 878
879 + // Refresh the bare repo from the canonical remote, then check the sha against
880 + // it here — before the trigger, so a bad sha is a synchronous 4xx rather than
881 + // an `accepted: true` and an exit 4 buried in the updater's journal.
882 + //
883 + // Only sandod writes this repo, and only by fetching the deploy branch from
884 + // the authenticated upstream, which is the whole basis of the updater's
885 + // provenance seal. Doing the fetch here keeps that true: no new writer.
886 + if let Some((bare, branch, upstream)) = self_update_source(&s) {
887 + // A fetch failure is non-fatal, matching the build path: the sha may
888 + // already be present from an earlier fetch, and the ancestry check below
889 + // is the real gate. What must not happen is proceeding *silently*.
890 + if let Some(upstream) = upstream.as_deref()
891 + && let Err(e) = crate::git::fetch_upstream(&bare, upstream, &branch).await
892 + {
893 + tracing::warn!(error = %e, upstream, "self-update upstream fetch failed; checking the sha against the current bare-repo state");
894 + }
895 + match crate::git::is_ancestor(&bare, sha.as_str(), &branch).await {
896 + Ok(true) => {}
897 + Ok(false) => {
898 + return Err(crate::error::Error::BadRequest(format!(
899 + "sha {sha} is not an ancestor of {branch} in {} — push it to the upstream remote first (the updater refuses any sha off the deploy branch)",
900 + bare.display(),
901 + )));
902 + }
903 + // Never block the update on our own pre-check being unable to answer:
904 + // the updater re-runs the identical check as root before it builds, so
905 + // the seal holds regardless. Losing the good error message is the only
906 + // cost of passing through here.
907 + Err(e) => {
908 + tracing::warn!(error = %e, "self-update ancestry pre-check could not run; deferring to the updater's own check");
909 + }
910 + }
911 + }
912 +
813 913 // Don't restart the controller out from under an in-flight server build — the
814 914 // restart would SIGKILL it mid-deploy. Hold the active_build slot across the
815 915 // trigger, not just the check: releasing it before triggering let a /rebuild
@@ -873,14 +973,19 @@
873 973 .find(|t| t.name == tier)
874 974 .ok_or(crate::error::Error::NotFound)?;
875 975
876 - let version_str: Option<String> =
877 - sqlx::query_scalar("SELECT current_version FROM tier_state WHERE app = ? AND tier = ?")
878 - .bind(&s.cfg.id)
879 - .bind(&target.name)
880 - .fetch_optional(&s.pool)
881 - .await
882 - .map_err(crate::error::Error::Db)?
883 - .flatten();
976 + // The build id comes back with the version so the row this writes carries the
977 + // identity every other gate row on this tier carries. Without it a
978 + // `manual_confirm` was the one gate `/state` could not scope to a build, and
979 + // it read as "not run" on a tier reporting per build.
980 + let confirmed: Option<(Option<String>, Option<i64>)> = sqlx::query_as(
981 + "SELECT current_version, current_build_id FROM tier_state WHERE app = ? AND tier = ?",
982 + )
983 + .bind(&s.cfg.id)
984 + .bind(&target.name)
985 + .fetch_optional(&s.pool)
986 + .await
987 + .map_err(crate::error::Error::Db)?;
988 + let (version_str, build_id) = confirmed.unwrap_or((None, None));
884 989 let version_str = version_str.ok_or_else(|| {
885 990 crate::error::Error::GateBlocked(format!(
886 991 "tier {tier} has no current_version; nothing to confirm"
@@ -897,11 +1002,12 @@
897 1002 let outcome_json = serde_json::to_string(&outcome)
898 1003 .map_err(|e| crate::error::Error::Other(anyhow::anyhow!(e)))?;
899 1004 sqlx::query(
900 - "INSERT INTO gate_runs (app, version, tier, gate_kind, started_at, finished_at, status, outcome_json)
901 - VALUES (?, ?, ?, 'manual_confirm', ?, ?, 'passed', ?)",
1005 + "INSERT INTO gate_runs (app, version, tier, gate_kind, started_at, finished_at, status, outcome_json, build_id)
1006 + VALUES (?, ?, ?, 'manual_confirm', ?, ?, 'passed', ?, ?)",
902 1007 )
903 1008 .bind(&s.cfg.id)
904 1009 .bind(&version).bind(&target.name).bind(&now).bind(&now).bind(&outcome_json)
1010 + .bind(build_id)
905 1011 .execute(&s.pool).await.map_err(crate::error::Error::Db)?;
906 1012
907 1013 tracing::info!(tier = %tier, version = %version, "manual_confirm recorded");
@@ -968,28 +1074,47 @@
968 1074 })))
969 1075 }
970 1076
1077 + /// The tail of one gate's log.
1078 + ///
1079 + /// `run` is the `build_runs.id` the gate ran for — the directory
1080 + /// [`GateCtx::log_scope`](crate::gates::GateCtx::log_scope) wrote under. It was
1081 + /// the version string until 2026-08-20, when every run of a version shared one
1082 + /// file and a reader of a rebuild got the previous attempt's output. Callers
1083 + /// follow the `log_ref` on the gate row rather than building this path, so both
1084 + /// scopes resolve and logs written under the old scheme stay readable at the
1085 + /// path their row records.
971 1086 async fn get_gate_log(
972 1087 State(s): State<AppState>,
973 - Path((version, gate)): Path<(String, String)>,
1088 + Path((run, gate)): Path<(String, String)>,
974 1089 ) -> Result<axum::response::Response> {
975 - // Guard against `..` / absolute paths — the version segment must be a single
1090 + // Guard against `..` / absolute paths — the scope segment must be a single
976 1091 // safe component. Without this, `GET /logs/..%2Fetc/passwd` would escape
977 1092 // logs_root.
978 1093 fn safe(seg: &str) -> bool {
979 1094 !seg.is_empty() && !seg.contains('/') && !seg.contains('\\') && seg != "." && seg != ".."
980 1095 }
981 - if !safe(&version) {
1096 + if !safe(&run) {
982 1097 return Err(crate::error::Error::NotFound);
983 1098 }
984 1099 // The gate segment is an allowlisted kind, not a free-form filename: parse it
985 - // to `GateKind` so only the five known `<kind>.log` files are reachable. This
986 - // closes `*.log` filename probing within a version dir (traversal was already
1100 + // to `GateKind` so only the known `<kind>.log` files are reachable. This
1101 + // closes `*.log` filename probing within a scope dir (traversal was already
987 1102 // blocked; this bounds the basename to the known set).
988 - let kind: crate::domain::GateKind = gate.parse().map_err(|_| crate::error::Error::NotFound)?;
1103 + //
1104 + // A trailing `.log` is accepted so a caller can append a gate row's `log_ref`
1105 + // to `/logs/` verbatim rather than taking it apart and putting it back
1106 + // together. Following the ref is the correct way to reach a log — building
1107 + // the path from the version reaches a different run's output on any version
1108 + // that was rebuilt — so the shape that gets it right should be the easy one.
1109 + let kind: crate::domain::GateKind = gate
1110 + .strip_suffix(".log")
1111 + .unwrap_or(&gate)
1112 + .parse()
1113 + .map_err(|_| crate::error::Error::NotFound)?;
989 1114 let path = s
990 1115 .cfg
991 1116 .logs_root
992 - .join(&version)
1117 + .join(&run)
993 1118 .join(format!("{}.log", kind.as_str()));
994 1119 // Bound how much a single request can pull into the daemon's memory: a
995 1120 // runaway gate log shouldn't be read whole. Past the cap we return the tail
@@ -1083,7 +1208,7 @@
1083 1208 #[cfg(test)]
1084 1209 mod tests {
1085 1210 use super::promotion::{
1086 - PromotedBuild, RollbackReport, rollback_deployed_nodes, unsatisfied_gates,
1211 + Evidence, PromotedBuild, RollbackReport, rollback_deployed_nodes, unsatisfied_gates,
1087 1212 };
1088 1213 use super::*;
1089 1214 use crate::config::AppConfig;
@@ -1380,6 +1505,57 @@
1380 1505 .unwrap();
1381 1506 }
1382 1507
1508 + /// A rebuild at an unchanged version must not move what a tier reports.
1509 + ///
1510 + /// `/state` keyed its gate rows on (tier, version), so two runs of one
1511 + /// version interleaved and each gate showed whichever had written it last.
1512 + /// The visible symptom was statuses that moved while nothing was touched:
1513 + /// two reads of the same tier a minute apart disagreed about whether
1514 + /// `hardening_test` had passed or never run. Runs 60-62 of mnw-server
1515 + /// 0.11.20, 2026-08-19.
1516 + #[tokio::test]
1517 + async fn state_reports_the_build_the_tier_is_running() {
1518 + let state = test_state().await;
1519 + seed(&state.pool, "a", "0.11.20").await;
1520 + let first = seed_build(&state.pool, "2a53c900", "0.11.20", "/rel/1111111111111111").await;
1521 + let second = seed_build(&state.pool, "adf56cd9", "0.11.20", "/rel/2222222222222222").await;
1522 + // The tier is running the first build, and it went green there.
1523 + insert_gate_build(&state.pool, "a", "0.11.20", "hardening_test", 1, first).await;
1524 + sqlx::query(
1525 + "UPDATE tier_state SET current_version = ?, current_build_id = ? WHERE tier = 'a'",
1526 + )
1527 + .bind("0.11.20")
1528 + .bind(first)
1529 + .execute(&state.pool)
1530 + .await
1531 + .unwrap();
1532 +
1533 + let before = state_view(&state).await.unwrap();
1534 + let gates_of = |v: &StateView| {
1535 + v.tiers
1536 + .iter()
1537 + .find(|t| t.name == "a")
1538 + .unwrap()
1539 + .gates
1540 + .iter()
1541 + .map(|g| (g.kind.clone(), g.status.clone()))
1542 + .collect::<Vec<_>>()
1543 + };
1544 + assert_eq!(
1545 + gates_of(&before),
1546 + vec![("hardening_test".to_string(), Some("passed".to_string()))],
1547 + );
1548 +
1549 + // A retry of the same version fails the same gate. The tier still runs
1550 + // the first build, so what it reports is unchanged.
1551 + insert_gate_build(&state.pool, "a", "0.11.20", "hardening_test", 0, second).await;
1552 + assert_eq!(
1553 + gates_of(&state_view(&state).await.unwrap()),
1554 + gates_of(&before),
1555 + "a sibling rebuild rewrote a tier it was never deployed to",
1556 + );
1557 + }
1558 +
1383 1559 // ---- unsatisfied_gates ----
1384 1560
1385 1561 fn tid(s: &str) -> crate::domain::TierId {
@@ -1396,8 +1572,11 @@
1396 1572 &crate::domain::AppId::default(),
1397 1573 &tid("host"),
1398 1574 &[],
1399 - "0.8.12",
1400 - &[],
1575 + &Evidence {
1576 + version: "0.8.12",
1577 + builds: &[],
1578 + tier_build: None,
1579 + },
1401 1580 false,
1402 1581 )
1403 1582 .await
@@ -1418,8 +1597,11 @@
1418 1597 &crate::domain::AppId::default(),
1419 1598 &tid("a"),
1420 1599 &[Gate::BootSmoke],
1421 - "0.8.12",
1422 - &[],
1600 + &Evidence {
1601 + version: "0.8.12",
1602 + builds: &[],
1603 + tier_build: None,
1604 + },
1423 1605 false,
1424 1606 )
1425 1607 .await
@@ -1438,8 +1620,11 @@
1438 1620 &crate::domain::AppId::default(),
1439 1621 &tid("host"),
1440 1622 &[Gate::CargoTest, Gate::BootSmoke],
1441 - "0.8.12",
1442 - &[],
1623 + &Evidence {
1624 + version: "0.8.12",
1625 + builds: &[],
1626 + tier_build: None,
1627 + },
1443 1628 false,
1444 1629 )
1445 1630 .await
@@ -1460,8 +1645,11 @@
1460 1645 &crate::domain::AppId::default(),
1461 1646 &tid("host"),
1462 1647 &[Gate::CargoTest],
1463 - "0.8.12",
1464 - &[],
1648 + &Evidence {
1649 + version: "0.8.12",
1650 + builds: &[],
1651 + tier_build: None,
1652 + },
1465 1653 false,
1466 1654 )
1467 1655 .await
@@ -1509,8 +1697,11 @@
1509 1697 &crate::domain::AppId::default(),
1510 1698 &tid("a"),
1511 1699 &[Gate::ManualConfirm],
1512 - "0.8.12",
1513 - &[],
1700 + &Evidence {
1701 + version: "0.8.12",
1702 + builds: &[],
1703 + tier_build: None,
1704 + },
1514 1705 false,
1515 1706 )
1516 1707 .await
@@ -1528,8 +1719,11 @@
1528 1719 &crate::domain::AppId::default(),
1529 1720 &tid("a"),
1530 1721 &[Gate::ManualConfirm],
1531 - "0.8.12",
1532 - &[],
1722 + &Evidence {
1723 + version: "0.8.12",
1724 + builds: &[],
1725 + tier_build: None,
1726 + },
1533 1727 false,
1534 1728 )
1535 1729 .await
@@ -1549,8 +1743,11 @@
1549 1743 &crate::domain::AppId::default(),
1550 1744 &tid("a"),
1551 1745 &[Gate::ManualConfirm],
1552 - "0.8.12",
1553 - &[],
1746 + &Evidence {
1747 + version: "0.8.12",
1748 + builds: &[],
1749 + tier_build: None,
1750 + },
1554 1751 false,
1555 1752 )
1556 1753 .await
@@ -1578,8 +1775,11 @@
1578 1775 &crate::domain::AppId::default(),
1579 1776 &tid("a"),
1580 1777 &gates,
1581 - "0.8.12",
1582 - &[],
1778 + &Evidence {
1779 + version: "0.8.12",
1780 + builds: &[],
1781 + tier_build: None,
1782 + },
1583 1783 false,
1584 1784 )
1585 1785 .await
@@ -1594,8 +1794,11 @@
1594 1794 &crate::domain::AppId::default(),
1595 1795 &tid("a"),
1596 1796 &gates,
1597 - "0.8.12",
1598 - &[],
1797 + &Evidence {
1798 + version: "0.8.12",
1799 + builds: &[],
1800 + tier_build: None,
1801 + },
1599 1802 true,
1600 1803 )
1601 1804 .await
@@ -1619,8 +1822,11 @@
1619 1822 &crate::domain::AppId::default(),
1620 1823 &tid("a"),
1621 1824 &[Gate::BurnIn { hours: 48 }],
1622 - "0.8.12",
1623 - &[],
Lines truncated
M sando/daemon/src/routes/promotion.rs +127 -24
@@ -188,13 +188,35 @@
188 188 });
189 189 }
190 190
191 + // The build the TARGET tier ends up running. The same as the source's for a
192 + // single-platform product, and its cross-architecture sibling when the two
193 + // tiers differ — promoting pom astra -> hetzner reads an aarch64 build from
194 + // the source and ships the x86_64 one. Every row this promote writes about
195 + // the target (its `deploys`, its post-deploy gate runs, its `tier_state`)
196 + // keys on this, so the next promote out of that tier resolves through
197 + // evidence for bytes that box actually ran.
198 + //
199 + // A tier whose nodes span platforms has no single current build, and NULL
200 + // says so rather than naming one of them arbitrarily. No such tier exists
201 + // today: pom's astra and hetzner tiers are one node each.
202 + let target_build_id = match promoted_builds.as_slice() {
203 + [only] => only.build_id,
204 + _ => None,
205 + };
206 +
191 207 let pending = unsatisfied_gates(
192 208 &s.pool,
193 209 &s.cfg.id,
194 210 &source.name,
195 211 &effective_gates,
196 - &version_str,
197 - &promoted_builds,
212 + &Evidence {
213 + version: &version_str,
214 + builds: &promoted_builds,
215 + // The source tier's own build: what its node_health / page_smoke rows
216 + // can possibly be about, which is not the same thing as what this
217 + // promote ships when the two tiers are different architectures.
218 + tier_build: build_id,
219 + },
198 220 body.hotfix,
199 221 )
200 222 .await?;
@@ -223,9 +245,7 @@
223 245 // sequential path. Track the nodes already flipped to the new version so
224 246 // a mid-rollout failure can roll them back (canary rollback).
225 247 let mut deployed: Vec<&crate::topology::Node> = Vec::new();
226 - // The build id is not read here: it did its work above, keying the gate
227 - // check to each bundle's own evidence. What ships is decided by `Placement`.
228 - for (node, node_bundle, node_bundle_platform, _) in &bundles {
248 + for (node, node_bundle, node_bundle_platform, node_build_id) in &bundles {
229 249 // The proof that these bytes belong on this box. Built before the
230 250 // deploy row is written, so a mismatch never becomes an `in_progress`
231 251 // deploy that has to be reconciled.
@@ -262,7 +282,12 @@
262 282 .bind(&version).bind(&target.name).bind(&node.name)
263 283 .bind(&started)
264 284 .bind(body.hotfix as i64).bind(body.reset_burn_in as i64)
265 - .bind(build_id)
285 + // The build THIS node received, which is not the source tier's when the
286 + // two are different architectures: promoting pom astra -> hetzner reads
287 + // an aarch64 build from the source and ships the x86_64 sibling. Binding
288 + // the source's id here recorded the wrong bytes against the deploy, and
289 + // `crate::reconcile` reads these rows.
290 + .bind(node_build_id)
266 291 .fetch_one(&s.pool).await.map_err(crate::error::Error::Db)?;
267 292 let result = crate::deploy::deploy_node(
268 293 executor.as_ref(),
@@ -468,8 +493,9 @@
468 493 nodes,
469 494 // These post-deploy gate rows are the evidence the NEXT promote (this
470 495 // tier -> the following one) resolves through, so they must carry the
471 - // build they vouch for.
472 - build_id,
496 + // build they vouch for — the one this tier just received, which is
497 + // not the source's across an architecture boundary.
498 + build_id: target_build_id,
473 499 // The hostname the public uses, straight from the tier. page_smoke
474 500 // has to request the site the way a visitor does; anything derived
475 501 // from a node would reach the origin and miss the CDN, which is the
@@ -514,9 +540,18 @@
514 540 // serialized against rollback and the host build path's advance.
515 541 // reset_burn_in on the *source* tier nulls its clock only when the operator
516 542 // explicitly asked.
517 - crate::runs::advance_tier(&s.pool, &s.cfg.id, target.name.as_str(), &version, build_id)
518 - .await
519 - .map_err(crate::error::Error::Db)?;
543 + // `target_build_id`, not the source's: recording the source's would have
544 + // left hetzner's `current_build_id` pointing at astra's aarch64 bundle,
545 + // i.e. at bytes that box cannot execute.
546 + crate::runs::advance_tier(
547 + &s.pool,
548 + &s.cfg.id,
549 + target.name.as_str(),
550 + &version,
551 + target_build_id,
552 + )
553 + .await
554 + .map_err(crate::error::Error::Db)?;
520 555
521 556 if body.reset_burn_in {
522 557 sqlx::query("UPDATE tier_state SET burn_in_started_at = NULL WHERE app = ? AND tier = ?")
@@ -625,6 +660,21 @@
625 660 pub build_id: Option<i64>,
626 661 }
627 662
663 + /// What a promote is judged against, in one value: the release it names, the
664 + /// artifacts it will ship, and the build the tier it is leaving is running.
665 + ///
666 + /// The three travel together because the gate check needs all three at once and
667 + /// they answer different questions. `version` is the label and the pre-identity
668 + /// fallback key. `builds` are the bytes going out, which artifact evidence is
669 + /// keyed on. `tier_build` is what the source tier is itself running, which
670 + /// post-deploy evidence is keyed on — and across an architecture boundary those
671 + /// last two are different builds. See [`unsatisfied_gates`].
672 + pub(super) struct Evidence<'a> {
673 + pub version: &'a str,
674 + pub builds: &'a [PromotedBuild],
675 + pub tier_build: Option<i64>,
676 + }
677 +
628 678 /// Which bytes each node gets, resolved before any node is touched.
629 679 ///
630 680 /// A single-platform product answers with `fallback` for every node, which is
@@ -949,16 +999,38 @@
949 999 /// on the tier's own clock (`tier_state.burn_in_started_at`), not on a build, so
950 1000 /// asking per build would ask the same question N times and answer it N times in
951 1001 /// the error.
1002 + ///
1003 + /// **So are `node_health` and `page_smoke`, and for a sharper reason: they are
1004 + /// evidence about this tier, not about the bytes leaving it.** `tier_build` is
1005 + /// the build this tier is itself running, and it is the only build those two can
1006 + /// ever have a row for. Asking them per shipped build is a category error that
1007 + /// made a cross-architecture promote structurally impossible: promoting pom
1008 + /// 0.4.3 astra -> hetzner ships the **x86_64** bundle, astra has only ever run
1009 + /// the **aarch64** one, and so `node_health` was looked up against a build astra
1010 + /// never saw and never could — astra is aarch64. Fail-closed reported it
1011 + /// unsatisfied, `/state` reported it passed (astra's own row, correctly), and the
1012 + /// promote refused twice with the gate satisfied by every reading the operator
1013 + /// had. It could not have been cleared by re-running anything.
1014 + ///
1015 + /// The artifact gates below stay per build and must: `cargo_test` on the aarch64
1016 + /// bundle says nothing about the x86_64 one, and each stands on its own intake.
1017 + /// What separates the two lists is whether the evidence travels with the bytes.
1018 + /// A single-platform product is unaffected either way — its source tier's build
1019 + /// *is* the build being shipped, so both readings ask the same question.
952 1020 pub(super) async fn unsatisfied_gates(
953 1021 pool: &sqlx::SqlitePool,
954 1022 app: &crate::domain::AppId,
955 1023 tier: &crate::domain::TierId,
956 1024 gates: &[crate::topology::Gate],
957 - version: &str,
958 - builds: &[PromotedBuild],
1025 + evidence: &Evidence<'_>,
959 1026 hotfix: bool,
960 1027 ) -> std::result::Result<Vec<String>, crate::error::Error> {
961 1028 use crate::topology::Gate;
1029 + let Evidence {
1030 + version,
1031 + builds,
1032 + tier_build,
1033 + } = *evidence;
962 1034 let mut bad = Vec::new();
963 1035 for gate in gates {
964 1036 let kind = gate.kind();
@@ -1020,12 +1092,47 @@
1020 1092 bad.push(kind.as_str().to_string());
1021 1093 }
1022 1094 }
1023 - // Build/post-deploy gates that leave a `gate_runs` row: the latest
1024 - // row for this (tier, version, kind) must be `passed`. Listed
1025 - // explicitly (no `_` catch-all) so adding a new `Gate` variant is a
1026 - // compile error here until its promotion semantics are decided —
1027 - // a transient-`blocked` kind silently falling into "needs a passed
1028 - // row" would be permanently unsatisfiable.
1095 + // Post-deploy tier gates: evidence that THIS tier is healthy running
1096 + // what it is running. Keyed on the tier's own build, once, because
1097 + // that is the only build a row here can ever exist for — see the note
1098 + // on `tier_build` above. `None` is the pre-identity tier, where the
1099 + // version-keyed lookup is the legacy behaviour.
1100 + Gate::NodeHealth | Gate::PageSmoke => {
1101 + let status: Option<String> = match tier_build {
1102 + Some(bid) => sqlx::query_scalar(
1103 + "SELECT status FROM gate_runs
1104 + WHERE app = ?1 AND tier = ?2 AND build_id = ?3 AND gate_kind = ?4
1105 + ORDER BY id DESC LIMIT 1",
1106 + )
1107 + .bind(app)
1108 + .bind(tier.as_str())
1109 + .bind(bid)
1110 + .bind(kind.as_str()),
1111 + None => sqlx::query_scalar(
1112 + "SELECT status FROM gate_runs
1113 + WHERE app = ?1 AND tier = ?2 AND version = ?3 AND gate_kind = ?4
1114 + ORDER BY id DESC LIMIT 1",
1115 + )
1116 + .bind(app)
1117 + .bind(tier.as_str())
1118 + .bind(version)
1119 + .bind(kind.as_str()),
1120 + }
1121 + .fetch_optional(pool)
1122 + .await
1123 + .map_err(crate::error::Error::Db)?
1124 + .flatten();
1125 + if status.as_deref() != Some("passed") {
1126 + bad.push(kind.as_str().to_string());
1127 + }
1128 + }
1129 + // Artifact gates: evidence about the bytes, which travels with them.
1130 + // The latest row for this (build, kind) must be `passed`, per build
1131 + // this promote ships. Listed explicitly (no `_` catch-all) so adding
1132 + // a new `Gate` variant is a compile error here until its promotion
1133 + // semantics are decided — a transient-`blocked` kind silently falling
1134 + // into "needs a passed row" would be permanently unsatisfiable, and
1135 + // a post-deploy kind landing here is the defect the arm above fixes.
1029 1136 Gate::CargoTest
1030 1137 | Gate::HardeningTest
1031 1138 | Gate::Clippy
@@ -1034,11 +1141,7 @@
1034 1141 | Gate::CargoDeny
1035 1142 | Gate::MigrationDryRun
1036 1143 | Gate::CodeSmoke
1037 - | Gate::BootSmoke
1038 - | Gate::NodeHealth
1039 - // Same evidence rule as node_health: a passed row for this tier's
1040 - // current build, or the promote out of here is refused.
1041 - | Gate::PageSmoke => {
1144 + | Gate::BootSmoke => {
1042 1145 // Latest row for this configured gate kind; NULL/missing/any
1043 1146 // non-'passed' status all count as unsatisfied (fail closed).
1044 1147 // Every build this promote ships has to show its own passed row.