Skip to main content

max / makenotwork

sando: unwedge the backup floor and block on stale dumps The plausibility floor is derived from the newest `backups` row, and a row is only written by a fetch that clears the floor. So a dump that legitimately halves seals the fetch shut: the reference can never advance, and every later fetch fails on the same stale number. Sando sat wedged from 2026-06-12 to 2026-07-27 after MNW started deleting expired tower_sessions rows and the prod dump dropped 43 MB -> 6 MB overnight. 45 days, no fresh backup. Add `force` to /backup/fetch: drop to the absolute floor for one fetch so the accepted dump becomes the new reference. It relaxes the size check only — `gzip -t` still refuses a truncated file, or the escape hatch would be a way to install a broken dump as the thing migration_dry_run restores. The second half is worse. migration_dry_run resolved its restore source with a presence check and never looked at the timestamp, so while the fetch failed closed the gate kept passing green against an ever-older schema — the exact signal it exists to raise, degrading silently. Block on age instead, bounded by the new `backup_max_age_hours` (default 48, so a daily fetch may miss one night). An unparsable fetched_at counts as stale: the column is daemon-written RFC 3339, so a value that will not parse means the row is untrustworthy, and a freshness check that shrugs at a timestamp it cannot read is not a freshness check. Carries the in-flight GateLog refactor in gates.rs, which was already in the tree and which the new gate tests are written against.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-27 19:56 UTC
Signed with PGP, not checked
Commit: be52980742e70d4c975d9becbe8e8eeb4011aa16
Parent: 6c8a6b4
7 files changed, +293 insertions, -178 deletions
M sando/README.md +2 -2
@@ -112,7 +112,7 @@
112 112 | `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. |
113 113 | `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. |
114 114 | `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. |
115 - | `migration_dry_run` | build | Migrations apply cleanly to a restored production dump. |
115 + | `migration_dry_run` | build | Migrations apply cleanly to a restored production dump. Blocks if the newest fetched dump is older than `backup_max_age_hours` (default 48) — a stale dump proves nothing about today's schema. |
116 116 | `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. |
117 117 | `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. |
118 118 | `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. |
@@ -237,7 +237,7 @@
237 237 | POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial` — the rollout landed, the tier cannot promote onward. |
238 238 | POST | `/rollback/{tier}` | — | Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. |
239 239 | POST | `/confirm/{tier}` | — | Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. |
240 - | POST | `/backup/fetch` | — | Pull the prod backup. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. |
240 + | POST | `/backup/fetch` | `{force?}` | Pull the prod backup. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. |
241 241 | GET | `/events` | — | WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). |
242 242
243 243 ## TUI
M sando/daemon/src/backup.rs +101 -6
@@ -96,6 +96,16 @@
96 96 /// valid gzip would pass `gzip -t` and the absolute floor, but not this. Every
97 97 /// row in `backups` is a previously-verified dump, so the last one is a sound
98 98 /// reference. Integer-halved at the call site.
99 + ///
100 + /// A dump *can* legitimately halve, though — a retention prune landing, a bloated
101 + /// table finally being swept. When that happens the floor is self-sealing: no new
102 + /// row is written unless a fetch clears it, so the reference can never advance and
103 + /// every later fetch fails on the same stale number. That is not hypothetical;
104 + /// Sando sat wedged from 2026-06-12 to 2026-07-27 after MNW started deleting
105 + /// expired `tower_sessions` rows and the dump dropped 43 MB -> 6 MB overnight.
106 + /// `force` (operator-supplied, via `POST /backup/fetch {"force":true}`) is the way
107 + /// out: it drops to the absolute floor for one fetch, so the accepted dump becomes
108 + /// the new reference. It never skips `gzip -t` — a truncated file is still refused.
99 109 const MIN_BACKUP_FRACTION_DENOM: i64 = 2;
100 110
101 111 /// Verify a freshly-downloaded backup before it is allowed to become the live
@@ -129,10 +139,16 @@
129 139 Ok(())
130 140 }
131 141
142 + /// Pull the configured prod dump into `topo.backup.local_path`.
143 + ///
144 + /// `force` re-baselines the plausibility floor: see `MIN_BACKUP_FRACTION_DENOM`.
145 + /// Pass `false` for anything automated — it is an operator escape hatch, not a
146 + /// retry strategy.
132 147 pub async fn fetch(
133 148 pool: &SqlitePool,
134 149 _cfg: &Arc<Config>,
135 150 topo: &Arc<Topology>,
151 + force: bool,
136 152 ) -> Result<FetchedBackup> {
137 153 let source = topo.backup.source.clone();
138 154 let local_path = topo.backup.local_path.clone();
@@ -158,9 +174,18 @@
158 174 sqlx::query_scalar("SELECT byte_size FROM backups ORDER BY fetched_at DESC LIMIT 1")
159 175 .fetch_optional(pool)
160 176 .await?;
161 - let min_bytes = last_size.map_or(MIN_BACKUP_BYTES, |s| {
162 - ((s / MIN_BACKUP_FRACTION_DENOM) as u64).max(MIN_BACKUP_BYTES)
163 - });
177 + let min_bytes = if force {
178 + tracing::warn!(
179 + last_verified_bytes = last_size,
180 + "force: re-baselining the backup plausibility floor to the absolute minimum; \
181 + this fetch's size becomes the new reference"
182 + );
183 + MIN_BACKUP_BYTES
184 + } else {
185 + last_size.map_or(MIN_BACKUP_BYTES, |s| {
186 + ((s / MIN_BACKUP_FRACTION_DENOM) as u64).max(MIN_BACKUP_BYTES)
187 + })
188 + };
164 189
165 190 let parsed = parse_source(&source)?;
166 191 let downloaded: Result<()> = async {
@@ -327,7 +352,7 @@
327 352 let pool = mem_pool().await;
328 353 let cfg = Arc::new(Config::for_tests());
329 354
330 - let fb = fetch(&pool, &cfg, &topo).await.unwrap();
355 + let fb = fetch(&pool, &cfg, &topo, false).await.unwrap();
331 356 assert!(dest.exists(), "live backup written");
332 357 assert!(
333 358 !dest.with_file_name("latest.sql.gz.partial").exists(),
@@ -364,7 +389,10 @@
364 389 .bind(dest.to_string_lossy().into_owned())
365 390 .execute(&pool).await.unwrap();
366 391
367 - let err = fetch(&pool, &cfg, &topo).await.unwrap_err().to_string();
392 + let err = fetch(&pool, &cfg, &topo, false)
393 + .await
394 + .unwrap_err()
395 + .to_string();
368 396 assert!(err.contains("implausibly small"), "{err}");
369 397 assert!(
370 398 !dest.exists(),
@@ -377,6 +405,73 @@
377 405 assert_eq!(count.0, 1, "the rejected fetch records no new row");
378 406 }
379 407
408 + #[tokio::test]
409 + async fn force_rebaselines_the_floor_after_a_legitimate_shrink() {
410 + // The wedge this exists for: the floor is derived from a row that only a
411 + // passing fetch can replace, so a dump that legitimately halves locks the
412 + // fetch out permanently. `force` accepts one undersized dump and makes it
413 + // the new reference, unwedging the next ordinary fetch.
414 + let tmp = tempfile::tempdir().unwrap();
415 + let src = tmp.path().join("src.sql.gz");
416 + write_valid_gz(&src).await;
417 + let dest = tmp.path().join("backups/latest.sql.gz");
418 + let topo = Arc::new(topo_with_backup(
419 + format!("file://{}", src.display()),
420 + dest.to_string_lossy().into_owned(),
421 + ));
422 + let pool = mem_pool().await;
423 + let cfg = Arc::new(Config::for_tests());
424 + sqlx::query("INSERT INTO backups (fetched_at, source, local_path, byte_size) VALUES (?, 'x', ?, 1000000)")
425 + .bind(Utc::now().to_rfc3339())
426 + .bind(dest.to_string_lossy().into_owned())
427 + .execute(&pool).await.unwrap();
428 +
429 + // Same dump the un-forced fetch rejects above.
430 + let fb = fetch(&pool, &cfg, &topo, true).await.unwrap();
431 + assert!(dest.exists(), "the forced dump becomes the live backup");
432 + let recorded = fb.byte_size.unwrap();
433 + assert!(
434 + recorded < 1_000_000,
435 + "the accepted dump really is the small one"
436 + );
437 +
438 + // The new row is now the reference, so the next fetch passes unforced.
439 + fetch(&pool, &cfg, &topo, false)
440 + .await
441 + .expect("floor re-baselined to the forced fetch's size");
442 + }
443 +
444 + #[tokio::test]
445 + async fn force_still_rejects_a_corrupt_gzip() {
446 + // `force` relaxes the size floor only. A truncated dump is refused either
447 + // way — otherwise the escape hatch would be a way to install a broken
448 + // backup as the thing migration_dry_run restores.
449 + let tmp = tempfile::tempdir().unwrap();
450 + let src = tmp.path().join("src.sql.gz");
451 + write_valid_gz(&src).await;
452 + let whole = tokio::fs::read(&src).await.unwrap();
453 + tokio::fs::write(&src, &whole[..whole.len() / 2])
454 + .await
455 + .unwrap();
456 + let dest = tmp.path().join("backups/latest.sql.gz");
457 + let topo = Arc::new(topo_with_backup(
458 + format!("file://{}", src.display()),
459 + dest.to_string_lossy().into_owned(),
460 + ));
461 + let pool = mem_pool().await;
462 + let cfg = Arc::new(Config::for_tests());
463 +
464 + let err = fetch(&pool, &cfg, &topo, true)
465 + .await
466 + .unwrap_err()
467 + .to_string();
468 + assert!(err.contains("gzip integrity check"), "{err}");
469 + assert!(
470 + !dest.exists(),
471 + "a corrupt dump never becomes the live backup"
472 + );
473 + }
474 +
380 475 #[tokio::test]
381 476 async fn fetch_rejects_truncated_gz_and_leaves_no_live_file() {
382 477 let tmp = tempfile::tempdir().unwrap();
@@ -400,7 +495,7 @@
400 495 let pool = mem_pool().await;
401 496 let cfg = Arc::new(Config::for_tests());
402 497
403 - let res = fetch(&pool, &cfg, &topo).await;
498 + let res = fetch(&pool, &cfg, &topo, false).await;
404 499 assert!(res.is_err(), "a truncated gzip must fail the fetch");
405 500 assert!(
406 501 !dest.exists(),
M sando/daemon/src/build.rs +2
@@ -696,6 +696,7 @@
696 696 scratch_db: true,
697 697 }],
698 698 frontend_builds: vec![],
699 + backup_max_age_hours: 48,
699 700 };
700 701
701 702 let topo = Topology {
@@ -775,6 +776,7 @@
775 776 companions: Vec::new(),
776 777 test_targets: vec![],
777 778 frontend_builds: vec![],
779 + backup_max_age_hours: 48,
778 780 }
779 781 }
780 782
M sando/daemon/src/config.rs +13
@@ -113,6 +113,15 @@
113 113 /// project that configures nothing keeps today's behavior.
114 114 #[serde(default, rename = "frontend_build")]
115 115 pub frontend_builds: Vec<FrontendBuild>,
116 + /// How old (hours) the fetched prod dump may be before `migration_dry_run`
117 + /// refuses to run against it. The gate restores whatever `backups` row is
118 + /// newest, and presence alone used to be the only check — so a fetch that
119 + /// stopped working left the gate passing green against an ever-older schema,
120 + /// which is the failure it exists to catch. Sando ran 45 days that way in
121 + /// June-July 2026. Default 48h: a daily fetch may miss one night without
122 + /// tripping this.
123 + #[serde(default = "default_backup_max_age_hours")]
124 + pub backup_max_age_hours: u32,
116 125 }
117 126
118 127 /// One npm project the `code_smoke` gate compiles.
@@ -278,6 +287,9 @@
278 287 fn default_gate_timeout_secs() -> u64 {
279 288 2400
280 289 }
290 + fn default_backup_max_age_hours() -> u32 {
291 + 48
292 + }
281 293
282 294 impl Config {
283 295 /// Primary binary — the one the systemd unit's ExecStart points at.
@@ -357,6 +369,7 @@
357 369 companions: Vec::new(),
358 370 test_targets: default_test_targets(),
359 371 frontend_builds: Vec::new(),
372 + backup_max_age_hours: default_backup_max_age_hours(),
360 373 }
361 374 }
362 375 }
M sando/daemon/src/gates.rs +149 -168
@@ -115,7 +115,7 @@
115 115 // doesn't orphan it.
116 116 Gate::MigrationDryRun => {
117 117 let ceiling = std::time::Duration::from_secs(ctx.cfg.gate_timeout_secs);
118 - match tokio::time::timeout(ceiling, migration_dry_run(ctx)).await {
118 + match tokio::time::timeout(ceiling, migration_dry_run(ctx, run_id)).await {
119 119 Ok(res) => res,
120 120 Err(_elapsed) => Ok(GateOutcome::failed(GateFailure::Timeout {
121 121 gate: GateKind::MigrationDryRun,
@@ -131,7 +131,7 @@
131 131 // it first (DROP IF EXISTS), same as migration_dry_run's scratch reset.
132 132 Gate::CodeSmoke => {
133 133 let ceiling = std::time::Duration::from_secs(ctx.cfg.gate_timeout_secs);
134 - match tokio::time::timeout(ceiling, code_smoke(ctx)).await {
134 + match tokio::time::timeout(ceiling, code_smoke(ctx, run_id)).await {
135 135 Ok(res) => res,
136 136 Err(_elapsed) => Ok(GateOutcome::failed(GateFailure::Timeout {
137 137 gate: GateKind::CodeSmoke,
@@ -828,77 +828,86 @@
828 828 cmd
829 829 }
830 830
831 - async fn migration_dry_run(ctx: &GateCtx) -> Result<GateOutcome> {
832 - let mut log_buf: Vec<u8> = Vec::new();
833 - let log_ref = LogRef::new(&ctx.version, GateKind::MigrationDryRun);
834 - let finish = |outcome: GateOutcome, buf: Vec<u8>| async move {
835 - persist_gate_log(ctx, GateKind::MigrationDryRun, &buf, &[]).await;
836 - outcome
837 - };
831 + async fn migration_dry_run(ctx: &GateCtx, run_id: GateRunId) -> Result<GateOutcome> {
832 + let log = GateLog::open(ctx, run_id, GateKind::MigrationDryRun).await;
833 + let outcome = migration_dry_run_inner(ctx, &log).await;
834 + log.close().await;
835 + outcome.map(|o| o.with_log_ref(LogRef::new(&ctx.version, GateKind::MigrationDryRun)))
836 + }
838 837
838 + /// The staged interior of [`migration_dry_run`], writing every step through the
839 + /// gate's live log. The caller owns the sink so it can flush it on every exit
840 + /// path, and attaches the `log_ref` once instead of at each return.
841 + async fn migration_dry_run_inner(ctx: &GateCtx, log: &GateLog) -> Result<GateOutcome> {
839 842 let Some(db_url) = ctx.cfg.scratch_db_url.as_deref() else {
840 - log_buf.extend_from_slice(b"scratch_db_url unset in daemon config\n");
841 - return Ok(finish(
842 - GateOutcome::blocked(GateBlocker::ScratchDbUrlUnset).with_log_ref(log_ref),
843 - log_buf,
844 - )
845 - .await);
843 + log.line("scratch_db_url unset in daemon config\n").await;
844 + return Ok(GateOutcome::blocked(GateBlocker::ScratchDbUrlUnset));
846 845 };
847 846
848 - let backup: Option<(String,)> =
849 - sqlx::query_as("SELECT local_path FROM backups ORDER BY id DESC LIMIT 1")
847 + let backup: Option<(String, String)> =
848 + sqlx::query_as("SELECT local_path, fetched_at FROM backups ORDER BY id DESC LIMIT 1")
850 849 .fetch_optional(&ctx.pool)
851 850 .await?;
852 - let Some((backup_path,)) = backup else {
853 - log_buf.extend_from_slice(b"no backup fetched; call /backup/fetch first\n");
854 - return Ok(finish(
855 - GateOutcome::blocked(GateBlocker::NoBackupAvailable).with_log_ref(log_ref),
856 - log_buf,
857 - )
858 - .await);
851 + let Some((backup_path, fetched_at)) = backup else {
852 + log.line("no backup fetched; call /backup/fetch first\n")
853 + .await;
854 + return Ok(GateOutcome::blocked(GateBlocker::NoBackupAvailable));
859 855 };
860 856
861 - log_buf.extend_from_slice(b"---- reset_scratch ----\n");
857 + // Presence is not freshness. A fetch that quietly stopped working leaves this
858 + // row in place, and restoring it dry-runs the migrations against a schema prod
859 + // has moved past — green, and worthless. Block on age instead. An unparsable
860 + // timestamp is treated as stale: this row is daemon-written RFC 3339, so a
861 + // value that will not parse means something is wrong, and failing closed on a
862 + // freshness check is the whole point.
863 + let age_hours = chrono::DateTime::parse_from_rfc3339(&fetched_at).map_or(i64::MAX, |t| {
864 + (Utc::now() - t.with_timezone(&Utc)).num_hours()
865 + });
866 + let max_age_hours = ctx.cfg.backup_max_age_hours;
867 + if age_hours > i64::from(max_age_hours) {
868 + let msg = format!(
869 + "backup {backup_path} was fetched {fetched_at} ({age_hours}h ago, max \
870 + {max_age_hours}h); re-run /backup/fetch\n"
871 + );
872 + log.line(&msg).await;
873 + return Ok(GateOutcome::blocked(GateBlocker::BackupStale {
874 + age_hours,
875 + max_age_hours,
876 + }));
877 + }
878 +
879 + log.line("---- reset_scratch ----\n").await;
862 880 if let Err(e) = reset_scratch(db_url, &ctx.cfg.scratch_owner_role).await {
863 881 let msg = format!("scratch reset: {e}");
864 - log_buf.extend_from_slice(msg.as_bytes());
865 - return Ok(finish(
866 - GateOutcome::failed(GateFailure::RestoreFailed { reason: msg }).with_log_ref(log_ref),
867 - log_buf,
868 - )
869 - .await);
882 + log.line(&msg).await;
883 + return Ok(GateOutcome::failed(GateFailure::RestoreFailed {
884 + reason: msg,
885 + }));
870 886 }
871 - log_buf.extend_from_slice(format!("---- restore_dump ({backup_path}) ----\n").as_bytes());
872 - if let Err(e) = restore_dump(db_url, &backup_path, &mut log_buf).await {
887 + log.line(&format!("---- restore_dump ({backup_path}) ----\n"))
888 + .await;
889 + if let Err(e) = restore_dump(db_url, &backup_path, log).await {
873 890 let msg = format!("restore: {e}");
874 - log_buf.extend_from_slice(msg.as_bytes());
875 - return Ok(finish(
876 - GateOutcome::failed(GateFailure::RestoreFailed { reason: msg }).with_log_ref(log_ref),
877 - log_buf,
878 - )
879 - .await);
891 + log.line(&msg).await;
892 + return Ok(GateOutcome::failed(GateFailure::RestoreFailed {
893 + reason: msg,
894 + }));
880 895 }
881 896
882 897 let migrations_dir = ctx.worktree.join("server").join("migrations");
883 - log_buf.extend_from_slice(b"---- run_migrator ----\n");
898 + log.line("---- run_migrator ----\n").await;
884 899 match run_migrator(db_url, &migrations_dir).await {
885 900 Ok(()) => {
886 - let detail = format!("restored {backup_path} + migrated");
887 - log_buf.extend_from_slice(detail.as_bytes());
888 - Ok(finish(
889 - GateOutcome::passed(PassNote::Migrated {
890 - backup_path: backup_path.clone(),
891 - })
892 - .with_log_ref(log_ref),
893 - log_buf,
894 - )
895 - .await)
901 + log.line(&format!("restored {backup_path} + migrated"))
902 + .await;
903 + Ok(GateOutcome::passed(PassNote::Migrated { backup_path }))
896 904 }
897 905 Err(e) => {
898 906 let err_s = e.to_string();
899 - log_buf.extend_from_slice(err_s.as_bytes());
900 - let failure = classify::classify_migration_error(&err_s, None);
901 - Ok(finish(GateOutcome::failed(failure).with_log_ref(log_ref), log_buf).await)
907 + log.line(&err_s).await;
908 + Ok(GateOutcome::failed(classify::classify_migration_error(
909 + &err_s, None,
910 + )))
902 911 }
903 912 }
904 913 }
@@ -1098,7 +1107,7 @@
1098 1107 }
1099 1108 }
1100 1109
1101 - async fn restore_dump(db_url: &str, dump: &str, log_buf: &mut Vec<u8>) -> Result<()> {
1110 + async fn restore_dump(db_url: &str, dump: &str, log: &GateLog) -> Result<()> {
1102 1111 // Split the password out of the URL and hand it to psql via PGPASSWORD, so it
1103 1112 // never lands in argv (visible in /proc/<pid>/cmdline to any local user).
1104 1113 // The sanitized URL — user/host/db, no secret — goes on the command line.
@@ -1114,13 +1123,13 @@
1114 1123 if let Some(pw) = password {
1115 1124 cmd.env("PGPASSWORD", pw);
1116 1125 }
1117 - let out = cmd.output().await?;
1118 - log_buf.extend_from_slice(&out.stdout);
1119 - log_buf.extend_from_slice(&out.stderr);
1126 + // Streamed, not `.output()`: a prod-sized restore runs for minutes, and
1127 + // psql's progress is the only thing an operator has to watch during it.
1128 + let (_stdout, stderr, status) = log.run(&mut cmd).await?;
1120 1129 anyhow::ensure!(
1121 - out.status.success(),
1130 + status.success(),
1122 1131 "restore failed: {}",
1123 - String::from_utf8_lossy(&out.stderr),
1132 + String::from_utf8_lossy(&stderr),
1124 1133 );
1125 1134 Ok(())
1126 1135 }
@@ -1231,21 +1240,20 @@
1231 1240 /// `HOST_URL` so config stays in dev mode (no CDN/S3/signing-secret prod
1232 1241 /// enforcement). The seed's host allowlist already admits `127.0.0.1`, and the
1233 1242 /// fresh DB trivially satisfies its no-real-users guard.
1234 - async fn code_smoke(ctx: &GateCtx) -> Result<GateOutcome> {
1235 - let log_ref = LogRef::new(&ctx.version, GateKind::CodeSmoke);
1236 - let mut log_buf: Vec<u8> = Vec::new();
1237 - let finish = |outcome: GateOutcome, buf: Vec<u8>| async move {
1238 - persist_gate_log(ctx, GateKind::CodeSmoke, &buf, &[]).await;
1239 - outcome
1240 - };
1243 + async fn code_smoke(ctx: &GateCtx, run_id: GateRunId) -> Result<GateOutcome> {
1244 + let log = GateLog::open(ctx, run_id, GateKind::CodeSmoke).await;
1245 + let outcome = code_smoke_inner(ctx, &log).await;
1246 + log.close().await;
1247 + outcome.map(|o| o.with_log_ref(LogRef::new(&ctx.version, GateKind::CodeSmoke)))
1248 + }
1241 1249
1250 + /// The staged interior of [`code_smoke`], writing every step through the gate's
1251 + /// live log. Same split as [`migration_dry_run_inner`]: the caller owns the sink
1252 + /// and attaches the `log_ref`.
1253 + async fn code_smoke_inner(ctx: &GateCtx, log: &GateLog) -> Result<GateOutcome> {
1242 1254 let Some(scratch_url) = ctx.cfg.scratch_db_url.as_deref() else {
1243 - log_buf.extend_from_slice(b"scratch_db_url unset in daemon config\n");
1244 - return Ok(finish(
1245 - GateOutcome::blocked(GateBlocker::ScratchDbUrlUnset).with_log_ref(log_ref),
1246 - log_buf,
1247 - )
1248 - .await);
1255 + log.line("scratch_db_url unset in daemon config\n").await;
1256 + return Ok(GateOutcome::blocked(GateBlocker::ScratchDbUrlUnset));
1249 1257 };
1250 1258
1251 1259 // The staged binary (set by build_and_run_host before gating). code_smoke
@@ -1257,14 +1265,9 @@
1257 1265 .fetch_optional(&ctx.pool)
1258 1266 .await?;
1259 1267 let Some((bin,)) = bin else {
1260 - return Ok(finish(
1261 - GateOutcome::blocked(GateBlocker::ArtifactMissing {
1262 - version: ctx.version.clone(),
1263 - })
1264 - .with_log_ref(log_ref),
1265 - log_buf,
1266 - )
1267 - .await);
1268 + return Ok(GateOutcome::blocked(GateBlocker::ArtifactMissing {
1269 + version: ctx.version.clone(),
1270 + }));
1268 1271 };
1269 1272
1270 1273 // Frontend builds, before anything else: they need no DB and no staged
@@ -1272,8 +1275,8 @@
1272 1275 // swallows (both MNW build scripts emit `cargo::warning` and succeed against
1273 1276 // a stale `static/dist/`). Failing here is what stops the deploy rsyncing
1274 1277 // the previous build's bundle.
1275 - if let Some(outcome) = code_smoke_frontends(ctx, &mut log_buf).await {
1276 - return Ok(finish(outcome.with_log_ref(log_ref), log_buf).await);
1278 + if let Some(outcome) = code_smoke_frontends(ctx, log).await {
1279 + return Ok(outcome);
1277 1280 }
1278 1281
1279 1282 // Docs integrity, first and cheapest: run the staged binary's DB-free
@@ -1282,8 +1285,8 @@
1282 1285 // in well under a second instead of after a full migrate+seed+boot, and a
1283 1286 // rotted link never reaches prod as a live 404. Collisions are reported by
1284 1287 // the check but do not fail it; only broken links do.
1285 - if let Some(outcome) = code_smoke_docs_check(ctx, &bin, &mut log_buf).await {
1286 - return Ok(finish(outcome.with_log_ref(log_ref), log_buf).await);
1288 + if let Some(outcome) = code_smoke_docs_check(ctx, &bin, log).await {
1289 + return Ok(outcome);
1287 1290 }
1288 1291
1289 1292 let dbname = code_smoke_db_name(&ctx.version);
@@ -1291,36 +1294,31 @@
1291 1294 let throwaway_url = pg_url_with_dbname(scratch_url, &dbname);
1292 1295
1293 1296 // Create the throwaway DB (dropping any stale one from a killed prior run).
1294 - log_buf.extend_from_slice(format!("---- createdb {dbname} ----\n").as_bytes());
1297 + log.line(&format!("---- createdb {dbname} ----\n")).await;
1295 1298 if let Err(e) = pg_create_db(&maintenance_url, &dbname).await {
1296 1299 let reason = format!("createdb {dbname}: {e}");
1297 - log_buf.extend_from_slice(reason.as_bytes());
1298 - return Ok(finish(
1299 - GateOutcome::failed(GateFailure::CodeSmokeSetup { reason }).with_log_ref(log_ref),
1300 - log_buf,
1301 - )
1302 - .await);
1300 + log.line(&reason).await;
1301 + return Ok(GateOutcome::failed(GateFailure::CodeSmokeSetup { reason }));
1303 1302 }
1304 1303
1305 1304 // Everything past createdb must drop the DB on the way out, pass or fail.
1306 - let outcome = code_smoke_body(ctx, &bin, &throwaway_url, &mut log_buf).await;
1305 + let outcome = code_smoke_body(ctx, &bin, &throwaway_url, log).await;
1307 1306
1308 - log_buf.extend_from_slice(format!("\n---- dropdb {dbname} ----\n").as_bytes());
1307 + log.line(&format!("\n---- dropdb {dbname} ----\n")).await;
1309 1308 if let Err(e) = pg_drop_db(&maintenance_url, &dbname).await {
1310 1309 // A teardown miss must not turn a passing gate red — log it and move on.
1311 1310 // The next run's createdb drops it first anyway.
1312 1311 tracing::warn!(error = %e, db = %dbname, "code_smoke: dropdb failed; next run will reclaim it");
1313 - log_buf.extend_from_slice(format!("dropdb warning (non-fatal): {e}").as_bytes());
1312 + log.line(&format!("dropdb warning (non-fatal): {e}")).await;
1314 1313 }
1315 1314
1316 - Ok(finish(outcome.with_log_ref(log_ref), log_buf).await)
1315 + Ok(outcome)
1317 1316 }
1318 1317
1319 1318 /// Compile every configured `frontend_build` in the worktree.
1320 1319 ///
1321 1320 /// Returns `Some(failed)` on the first project that does not build; `None` when
1322 - /// all of them do (or none are configured). Output is appended to `log_buf`
1323 - /// either way.
1321 + /// all of them do (or none are configured). Output streams to `log` either way.
1324 1322 ///
1325 1323 /// `npm ci` runs only when `node_modules` is absent. Usually it is not: the app
1326 1324 /// build script installed it during the `cargo build` that produced the artifact
@@ -1332,23 +1330,23 @@
1332 1330 ///
1333 1331 /// Unlike the app build scripts, nothing here is best-effort. That asymmetry is
1334 1332 /// the point of the gate.
1335 - async fn code_smoke_frontends(ctx: &GateCtx, log_buf: &mut Vec<u8>) -> Option<GateOutcome> {
1333 + async fn code_smoke_frontends(ctx: &GateCtx, log: &GateLog) -> Option<GateOutcome> {
1336 1334 for fe in &ctx.cfg.frontend_builds {
1337 1335 let dir = ctx.worktree.join(&fe.dir);
1338 1336 let label = fe.dir.display().to_string();
1339 - log_buf.extend_from_slice(format!("---- frontend build ({label}) ----\n").as_bytes());
1337 + log.line(&format!("---- frontend build ({label}) ----\n"))
1338 + .await;
1340 1339
1341 1340 if !dir.is_dir() {
1342 1341 // An older sha predating the frontend, mid-bisect. Skipping keeps
1343 1342 // sando able to rebuild history; the log says so out loud.
1344 - log_buf.extend_from_slice(
1345 - format!("{label} absent from this worktree; skipping\n").as_bytes(),
1346 - );
1343 + log.line(&format!("{label} absent from this worktree; skipping\n"))
1344 + .await;
1347 1345 continue;
1348 1346 }
1349 1347
1350 1348 if !dir.join("node_modules").is_dir()
1351 - && let Some(outcome) = run_npm(&dir, &label, &["ci"], "npm ci", ctx, log_buf).await
1349 + && let Some(outcome) = run_npm(&dir, &label, &["ci"], "npm ci", ctx, log).await
1352 1350 {
1353 1351 return Some(outcome);
1354 1352 }
@@ -1359,7 +1357,7 @@
1359 1357 &["run", &fe.script],
1360 1358 &format!("npm run {}", fe.script),
1361 1359 ctx,
1362 - log_buf,
1360 + log,
1363 1361 )
1364 1362 .await
1365 1363 {
@@ -1379,92 +1377,83 @@
1379 1377 args: &[&str],
1380 1378 what: &str,
1381 1379 ctx: &GateCtx,
1382 - log_buf: &mut Vec<u8>,
1380 + log: &GateLog,
1383 1381 ) -> Option<GateOutcome> {
1384 - log_buf.extend_from_slice(format!("$ {what}\n").as_bytes());
1382 + log.line(&format!("$ {what}\n")).await;
1385 1383 let mut cmd = tokio::process::Command::new("npm");
1386 - cmd.args(args)
1387 - .current_dir(dir)
1388 - .stdout(std::process::Stdio::piped())
1389 - .stderr(std::process::Stdio::piped())
1390 - .kill_on_drop(true);
1384 + cmd.args(args).current_dir(dir).kill_on_drop(true);
1391 1385 let ceiling = std::time::Duration::from_secs(ctx.cfg.gate_timeout_secs);
1392 - let out = match tokio::time::timeout(ceiling, cmd.output()).await {
1393 - Ok(Ok(out)) => out,
1386 + // On the timeout branch the whole `run` future is dropped, which drops the
1387 + // child; `kill_on_drop` is what turns that into an actual kill.
1388 + let status = match tokio::time::timeout(ceiling, log.run(&mut cmd)).await {
1389 + Ok(Ok((_stdout, _stderr, status))) => status,
1394 1390 Ok(Err(e)) => {
1395 1391 // A missing `npm` lands here. Fatal, not skipped: a build host
1396 1392 // without Node cannot produce the bundle the release serves, and
1397 1393 // silently passing is how the stale bundle shipped in the first place.
1398 - log_buf.extend_from_slice(format!("{what} could not be spawned: {e}\n").as_bytes());
1394 + log.line(&format!("{what} could not be spawned: {e}\n"))
1395 + .await;
1399 1396 return Some(GateOutcome::failed(GateFailure::SpawnFailed {
1400 1397 message: format!("{what} in {label}: {e}"),
1401 1398 }));
1402 1399 }
1403 1400 Err(_elapsed) => {
1404 - log_buf.extend_from_slice(
1405 - format!("{what} timed out after {}s\n", ctx.cfg.gate_timeout_secs).as_bytes(),
1406 - );
1401 + log.line(&format!(
1402 + "{what} timed out after {}s\n",
1403 + ctx.cfg.gate_timeout_secs
1404 + ))
1405 + .await;
1407 1406 return Some(GateOutcome::failed(GateFailure::CodeSmokeFrontend {
1408 1407 dir: label.to_string(),
1409 1408 exit_code: None,
1410 1409 }));
1411 1410 }
1412 1411 };
1413 - log_buf.extend_from_slice(&out.stdout);
1414 - log_buf.extend_from_slice(&out.stderr);
1415 - if out.status.success() {
1412 + if status.success() {
1416 1413 return None;
1417 1414 }
1418 1415 Some(GateOutcome::failed(GateFailure::CodeSmokeFrontend {
1419 1416 dir: label.to_string(),
1420 - exit_code: out.status.code(),
1417 + exit_code: status.code(),
1421 1418 }))
1422 1419 }
1423 1420
1424 1421 /// Run the staged binary's DB-free docs integrity check (`MNW_CHECK_DOCS=1`).
1425 1422 ///
1426 1423 /// Returns `Some(failed)` if the check reports broken links, cannot be spawned,
1427 - /// or overruns its ceiling; `None` when the docs are clean. Output is appended
1428 - /// to `log_buf` either way. The 60s ceiling backstops the case where the staged
1429 - /// binary predates the flag and would fall through to a normal (DB-needing)
1430 - /// boot and hang.
1431 - async fn code_smoke_docs_check(
1432 - ctx: &GateCtx,
1433 - bin: &str,
1434 - log_buf: &mut Vec<u8>,
1435 - ) -> Option<GateOutcome> {
1424 + /// or overruns its ceiling; `None` when the docs are clean. Output streams to
1425 + /// `log` either way. The 60s ceiling backstops the case where the staged binary
1426 + /// predates the flag and would fall through to a normal (DB-needing) boot and
1427 + /// hang.
1428 + async fn code_smoke_docs_check(ctx: &GateCtx, bin: &str, log: &GateLog) -> Option<GateOutcome> {
1436 1429 let server_dir = ctx.worktree.join("server");
1437 - log_buf.extend_from_slice(b"---- docs check (MNW_CHECK_DOCS) ----\n");
1430 + log.line("---- docs check (MNW_CHECK_DOCS) ----\n").await;
1438 1431 let mut cmd = tokio::process::Command::new(bin);
1439 1432 cmd.env("MNW_CHECK_DOCS", "1")
1440 1433 .current_dir(&server_dir)
1441 - .stdout(std::process::Stdio::piped())
1442 - .stderr(std::process::Stdio::piped())
1443 1434 .kill_on_drop(true);
1444 - let out = match tokio::time::timeout(std::time::Duration::from_mins(1), cmd.output()).await {
1445 - Ok(Ok(out)) => out,
1446 - Ok(Err(e)) => {
1447 - log_buf.extend_from_slice(format!("docs check spawn failed: {e}\n").as_bytes());
1448 - return Some(GateOutcome::failed(GateFailure::SpawnFailed {
1449 - message: e.to_string(),
1450 - }));
1451 - }
1452 - Err(_elapsed) => {
1453 - let reason =
1454 - "docs check timed out after 60s (staged binary may predate MNW_CHECK_DOCS)"
1455 - .to_string();
1456 - log_buf.extend_from_slice(reason.as_bytes());
1457 - log_buf.push(b'\n');
1458 - return Some(GateOutcome::failed(GateFailure::CodeSmokeSetup { reason }));
1459 - }
1460 - };
1461 - log_buf.extend_from_slice(&out.stdout);
1462 - log_buf.extend_from_slice(&out.stderr);
1463 - if out.status.success() {
1435 + let (stdout, _stderr, status) =
1436 + match tokio::time::timeout(std::time::Duration::from_mins(1), log.run(&mut cmd)).await {
1437 + Ok(Ok(out)) => out,
1438 + Ok(Err(e)) => {
1439 + log.line(&format!("docs check spawn failed: {e}\n")).await;
1440 + return Some(GateOutcome::failed(GateFailure::SpawnFailed {
1441 + message: e.to_string(),
1442 + }));
1443 + }
1444 + Err(_elapsed) => {
1445 + let reason =
1446 + "docs check timed out after 60s (staged binary may predate MNW_CHECK_DOCS)"
1447 + .to_string();
1448 + log.line(&format!("{reason}\n")).await;
1449 + return Some(GateOutcome::failed(GateFailure::CodeSmokeSetup { reason }));
1450 + }
1451 + };
1452 + if status.success() {
1464 1453 return None;
1465 1454 }
1466 1455 Some(GateOutcome::failed(GateFailure::CodeSmokeDocs {
1467 - broken: parse_check_docs_broken_count(&out.stdout),
1456 + broken: parse_check_docs_broken_count(&stdout),
1468 1457 }))
1469 1458 }
1470 1459
@@ -1488,40 +1477,30 @@
1488 1477 /// The createdb-to-dropdb interior of `code_smoke`: migrate+seed, then boot and
1489 1478 /// probe. Returns the outcome without a `log_ref` (the caller attaches it after
1490 1479 /// teardown). Never returns `Err` — spawn/child failures map to typed outcomes.
1491 - async fn code_smoke_body(
1492 - ctx: &GateCtx,
1493 - bin: &str,
1494 - db_url: &str,
1495 - log_buf: &mut Vec<u8>,
1496 - ) -> GateOutcome {
1480 + async fn code_smoke_body(ctx: &GateCtx, bin: &str, db_url: &str, log: &GateLog) -> GateOutcome {
1497 1481 let server_dir = ctx.worktree.join("server");
1498 1482
1499 1483 // Phase 1: migrate-from-scratch + seed. `--seed-examples` loads config,
1500 1484 // connects, runs migrations against the empty DB, seeds the catalog, exits.
1501 1485 // A non-zero exit here is the "code is unsound" signal (broken migration,
1502 1486 // seed error, or config-load failure).
1503 - log_buf.extend_from_slice(b"---- migrate + seed (--seed-examples) ----\n");
1487 + log.line("---- migrate + seed (--seed-examples) ----\n")
1488 + .await;
1504 1489 let mut seed_cmd = tokio::process::Command::new(bin);
1505 1490 seed_cmd.arg("--seed-examples").current_dir(&server_dir);
1506 1491 code_smoke_env(&mut seed_cmd, ctx, db_url);
1507 - seed_cmd
1508 - .env("ALLOW_EXAMPLE_SEED", "1")
1509 - .stdout(std::process::Stdio::piped())
1510 - .stderr(std::process::Stdio::piped())
1511 - .kill_on_drop(true);
1512 - let seed_out = match seed_cmd.output().await {
1513 - Ok(o) => o,
1492 + seed_cmd.env("ALLOW_EXAMPLE_SEED", "1").kill_on_drop(true);
1493 + let seed_status = match log.run(&mut seed_cmd).await {
1494 + Ok((_stdout, _stderr, status)) => status,
1514 1495 Err(e) => {
1515 1496 return GateOutcome::failed(GateFailure::SpawnFailed {
1516 1497 message: e.to_string(),
1517 1498 });
1518 1499 }
Lines truncated
M sando/daemon/src/outcome.rs +9
@@ -142,6 +142,11 @@
142 142 AwaitingOperatorConfirmation,
143 143 /// `migration_dry_run`: no row in `backups` to restore from.
144 144 NoBackupAvailable,
145 + /// `migration_dry_run`: the newest `backups` row is older than
146 + /// `cfg.backup_max_age_hours`. Restoring it would dry-run the migrations
147 + /// against a schema prod has since moved past, which passes green while
148 + /// proving nothing — so the gate blocks instead.
149 + BackupStale { age_hours: i64, max_age_hours: u32 },
145 150 /// `migration_dry_run` / `boot_smoke` / `cargo_test`: daemon config
146 151 /// has no `scratch_db_url`.
147 152 ScratchDbUrlUnset,
@@ -162,6 +167,10 @@
162 167 } => format!("{hours_remaining} hours remaining of {hours_total}"),
163 168 GateBlocker::AwaitingOperatorConfirmation => "waiting on operator confirmation".into(),
164 169 GateBlocker::NoBackupAvailable => "no backup fetched; call /backup/fetch first".into(),
170 + GateBlocker::BackupStale {
171 + age_hours,
172 + max_age_hours,
173 + } => format!("backup is {age_hours}h old (max {max_age_hours}h); re-run /backup/fetch"),
165 174 GateBlocker::ScratchDbUrlUnset => "scratch_db_url unset in daemon config".into(),
166 175 GateBlocker::ArtifactMissing { version } => {
167 176 format!("no artifact for version {version}")
M sando/daemon/src/routes/mod.rs +17 -2
@@ -707,8 +707,22 @@
707 707 ))
708 708 }
709 709
710 - async fn backup_fetch(State(s): State<AppState>) -> Result<Json<serde_json::Value>> {
711 - let fb = crate::backup::fetch(&s.pool, &s.cfg, &s.topo)
710 + #[derive(Deserialize, Default)]
711 + struct BackupFetchBody {
712 + /// Accept a dump that falls below the plausibility floor, re-baselining it to
713 + /// this fetch's size. For the case where the source legitimately shrank and
714 + /// the floor has wedged itself; the `gzip -t` integrity check still applies.
715 + /// Operator-only — never set this from a timer.
716 + #[serde(default)]
717 + force: bool,
718 + }
719 +
720 + async fn backup_fetch(
721 + State(s): State<AppState>,
722 + body: Option<Json<BackupFetchBody>>,
723 + ) -> Result<Json<serde_json::Value>> {
724 + let body = body.map(|Json(b)| b).unwrap_or_default();
725 + let fb = crate::backup::fetch(&s.pool, &s.cfg, &s.topo, body.force)
712 726 .await
713 727 .map_err(crate::error::Error::Other)?;
714 728 crate::events::emit(
@@ -934,6 +948,7 @@
934 948 scratch_db: true,
935 949 }],
936 950 frontend_builds: vec![],
951 + backup_max_age_hours: 48,
937 952 }
938 953 }
939 954