Skip to main content

max / makenotwork

Give code_smoke a cache that survives PrivateTmp, and stop misreading a bad fetch Build 60 failed code_smoke with seventeen digest mismatches, every one a metmuseum.org URL, and parked 0.11.20 on a red run. The pins were never wrong: all 34 cached assets on fw13 still match this manifest byte for byte, and a cold fetch of all 34 from the live origins matches too. What was wrong is that the gate had no cache. SEED_MEDIA_CACHE defaults under $TMPDIR and sandod runs with PrivateTmp=true, so every build opened on an empty /tmp and re-downloaded all 34 assets, while the same fetch from a shell on the same machine hit a months-old warm cache and never touched the network. That is the whole discrepancy: one machine asking 34 times per build, the other not asking. A bad minute at the Met's CDN then reads as "the asset changed at the source" and sends whoever is holding the build off to re-check a licence. Two halves, neither of which repins anything: - sando grows `code_smoke_env`, arbitrary KEY=VALUE a product hands its own binary, so the daemon stays product-agnostic. The fixed set is applied after it and wins on collision: nothing in a config file may point a smoke run off its throwaway DB. MNW sets SEED_MEDIA_CACHE=/srv/sando/seed-media-cache. - the fetch retries a mismatch three times before believing it, and refuses a response that is a document where media was declared. The test is "did we get a page about the asset" rather than an equality check on content type, because Wikimedia serves application/ogg for files declared audio/ogg and an equality check fails all three of them. A 404 stays fatal and fails on the first attempt; a digest that never matches still fails, and still says to re-check the licence.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-20 02:08 UTC
Signed with PGP, not checked
Commit: 88a951f783f5f7268096d029b1a9b6e75aec2a3e
Parent: 1ede82f
7 files changed, +407 insertions, -16 deletions
M sando/daemon/sando-daemon.toml +17
@@ -12,6 +12,7 @@
12 12 # the previous sha's compiled deps instead of clean-compiling each fresh
13 13 # worktree. Safe because builds are serialized. Omit for per-worktree target/.
14 14 cargo_target_dir = "./cargo-target"
15 +
15 16 # Dropped and recreated on every migration_dry_run. Leave unset to skip.
16 17 # The role must be SUPERUSER on the scratch cluster: the gates reset it, seed the
17 18 # dump's owner role into it, and drop stale mnw_test_* clones left by a killed
@@ -176,3 +177,19 @@
176 177 # Sando gates and promotes what arrives. See sando-pom.toml.
177 178 [app.pom]
178 179 config = "sando-pom.toml"
180 +
181 + # Extra env for the code_smoke gate's binary. The fixed set (DATABASE_URL, HOST,
182 + # PORT, ...) overrides anything here, so this cannot redirect the gate.
183 + #
184 + # SEED_MEDIA_CACHE: the example seed fetches 34 third-party media assets and
185 + # caches them by id, trusting a cached file only when it matches the digest the
186 + # manifest pins. That cache defaults to `$TMPDIR/mnw-seed-media`, and sandod runs
187 + # with `PrivateTmp=true`, so every build got an empty `/tmp` and re-downloaded
188 + # all 34 from metmuseum.org and Wikimedia. Build 60 (2026-08-19) is what that
189 + # costs: seventeen of them came back as something other than the pinned bytes in
190 + # one burst, code_smoke went red, and 0.11.20 was parked on a release that had
191 + # nothing wrong with it. A persistent directory makes the cache work as intended
192 + # — the network is touched only for an asset not already held at its pinned
193 + # digest, so a bad minute at someone else's CDN cannot fail a build.
194 + [code_smoke_env]
195 + SEED_MEDIA_CACHE = "/srv/sando/seed-media-cache"
M sando/daemon/src/build.rs +2
@@ -956,6 +956,7 @@
956 956 let cfg = AppConfig {
957 957 page_smoke_cmd: None,
958 958 platform: None,
959 + code_smoke_env: Default::default(),
959 960 id: crate::domain::AppId::default(),
960 961 topology_path: PathBuf::from("/tmp/test-sando.toml"),
961 962 build_host: Some("test-host".into()),
@@ -1276,6 +1277,7 @@
1276 1277 AppConfig {
1277 1278 page_smoke_cmd: None,
1278 1279 platform: None,
1280 + code_smoke_env: Default::default(),
1279 1281 id: crate::domain::AppId::default(),
1280 1282 topology_path: PathBuf::from("/tmp/test-sando.toml"),
1281 1283 build_host: Some("test-host".into()),
M sando/daemon/src/config.rs +32
@@ -160,6 +160,25 @@
160 160 /// behavior). Cargo creates the dir if absent.
161 161 #[serde(default)]
162 162 pub cargo_target_dir: Option<PathBuf>,
163 + /// Extra environment the `code_smoke` gate hands the binary it just built,
164 + /// on top of the fixed set every invocation shares. Project-supplied, so
165 + /// sando itself stays product-agnostic: the daemon knows it is passing
166 + /// `KEY=VALUE` through and nothing about what any key means.
167 + ///
168 + /// The fixed set wins on a collision. `DATABASE_URL`, `HOST`, `PORT` and
169 + /// the rest are what point the gate at its own throwaway DB and loopback
170 + /// port, and a typo here must not be able to aim a smoke run at something
171 + /// real.
172 + ///
173 + /// What it exists for: a gate that reaches the network on every run is a
174 + /// gate that fails on someone else's outage. MNW's example seed fetches 34
175 + /// third-party media assets and caches them by digest, but the daemon runs
176 + /// under `PrivateTmp=true`, so the cache's default home in `/tmp` is a
177 + /// fresh empty directory for every build and the gate re-downloads all 34
178 + /// every time. Pointing `SEED_MEDIA_CACHE` at a persistent directory makes
179 + /// the cache do the job it was written for.
180 + #[serde(default)]
181 + pub code_smoke_env: BTreeMap<String, String>,
163 182 /// Non-binary contents to stage into each release dir alongside
164 183 /// `bin_names`. Each entry copies `worktree/<src>` into
165 184 /// `<release>/<dst>`. `required=false` makes a missing source a warn
@@ -665,6 +684,7 @@
665 684 logs_root: PathBuf::from("/tmp/sando-test-logs"),
666 685 release_contents: Vec::new(),
667 686 cargo_target_dir: None,
687 + code_smoke_env: BTreeMap::new(),
668 688 gate_timeout_secs: default_gate_timeout_secs(),
669 689 companions: Vec::new(),
670 690 test_targets: default_test_targets(),
@@ -970,6 +990,18 @@
970 990 .any(|t| t.dir == std::path::Path::new("mnw-cli")),
971 991 "the companion that installs onto prod-1 must be gated",
972 992 );
993 + // serde ignores unknown keys, so a misspelled entry here is not a parse
994 + // error — it is a setting that silently does nothing, and code_smoke
995 + // would go on re-downloading 34 media assets per build with nothing
996 + // saying so. Assert the key by name.
997 + assert_eq!(
998 + cfg.code_smoke_env
999 + .get("SEED_MEDIA_CACHE")
1000 + .map(String::as_str),
1001 + Some("/srv/sando/seed-media-cache"),
1002 + "the seed's media cache must point somewhere that survives \
1003 + PrivateTmp=true",
1004 + );
973 1005 // Both crates serve JS their build scripts compile best-effort, so an
974 1006 // unlisted frontend is an ungated bundle.
975 1007 for dir in ["server/frontend", "multithreaded/frontend"] {
M sando/daemon/src/gates.rs +48
@@ -1836,6 +1836,11 @@
1836 1836 /// migrate, seed, boot and serve; whether a given deployment's env is complete
1837 1837 /// is the `config_check_env_file` guard's job, on the node, against that node's
1838 1838 /// real env file.
1839 + ///
1840 + /// `app.code_smoke_env` is prepended to all of this, for vars a product needs
1841 + /// that sando has no business knowing about. The fixed set overwrites it on a
1842 + /// collision, so nothing in a config file can redirect the gate off its own
1843 + /// throwaway DB.
1839 1844 fn code_smoke_env(cmd: &mut tokio::process::Command, ctx: &GateCtx, db_url: &str) {
1840 1845 // `localhost`, not `127.0.0.1`, and the distinction is load-bearing. The
1841 1846 // server derives its WebAuthn relying-party id from HOST_URL's host, and
@@ -1848,6 +1853,10 @@
1848 1853 // HOST stays 127.0.0.1: that is the bind address, and the gate probes the
1849 1854 // loopback address directly, so only the advertised origin changes.
1850 1855 let origin = format!("http://localhost:{}", ctx.cfg.code_smoke_port);
1856 + // Project-supplied extras go on first so the fixed set below overwrites any
1857 + // key they collide on: what points this run at its throwaway DB and its
1858 + // loopback port is not negotiable from a config file.
1859 + cmd.envs(&ctx.cfg.code_smoke_env);
1851 1860 cmd.env("DATABASE_URL", db_url)
1852 1861 .env("HOST", "127.0.0.1")
1853 1862 .env("PORT", ctx.cfg.code_smoke_port.to_string())
@@ -3828,6 +3837,45 @@
3828 3837 assert!(set["SIGNING_SECRET"].len() >= 32);
3829 3838 }
3830 3839
3840 + #[tokio::test]
3841 + async fn code_smoke_env_passes_extras_through_but_never_lets_them_win() {
3842 + // The pass-through exists so a product can hand its own binary a var
3843 + // sando has no business knowing about (MNW points SEED_MEDIA_CACHE at a
3844 + // persistent dir, because PrivateTmp=true made the seed's media cache
3845 + // cold on every build). What it must never become is a way to aim a
3846 + // smoke run at a real database.
3847 + let mut cfg = crate::config::AppConfig::for_tests();
3848 + cfg.code_smoke_env = [
3849 + (
3850 + "SEED_MEDIA_CACHE".to_string(),
3851 + "/srv/sando/seed".to_string(),
3852 + ),
3853 + (
3854 + "DATABASE_URL".to_string(),
3855 + "postgres://prod-1/makenotwork".to_string(),
3856 + ),
3857 + ]
3858 + .into_iter()
3859 + .collect();
3860 + let mut ctx = resolving_ctx("/w/abc", &[]);
3861 + ctx.cfg = std::sync::Arc::new(cfg);
3862 +
3863 + let mut cmd = tokio::process::Command::new("true");
3864 + code_smoke_env(&mut cmd, &ctx, "postgres:///throwaway");
3865 + let set: std::collections::HashMap<String, String> = cmd
3866 + .as_std()
3867 + .get_envs()
3868 + .filter_map(|(k, v)| Some((k.to_str()?.to_string(), v?.to_str()?.to_string())))
3869 + .collect();
3870 +
3871 + assert_eq!(set["SEED_MEDIA_CACHE"], "/srv/sando/seed");
3872 + assert_eq!(
3873 + set["DATABASE_URL"], "postgres:///throwaway",
3874 + "the fixed set must overwrite a colliding extra, or a config typo \
3875 + could point code_smoke at a real database",
3876 + );
3877 + }
3878 +
3831 3879 #[test]
3832 3880 fn code_smoke_db_name_sanitizes_and_caps() {
3833 3881 assert_eq!(
M server/src/seed/manifest.rs +287 -16
@@ -50,6 +50,16 @@
50 50 /// multi-gigabyte URL in the manifest is a mistake, not a big file.
51 51 const MAX_ASSET_BYTES: u64 = 64 * 1024 * 1024;
52 52
53 + /// How many times to fetch one asset before believing a digest mismatch.
54 + ///
55 + /// A mismatch has two causes and they want opposite responses: the asset really
56 + /// changed at the source (stop, re-check the licence), or this particular
57 + /// response was not the asset (retry, and it comes back right). Build 60 on
58 + /// 2026-08-19 was the second: seventeen pinned assets mismatched in one burst
59 + /// and every one of them re-fetched byte-identical to its pin afterwards, from
60 + /// the same machine. One retry separates the two cheaply.
61 + const FETCH_ATTEMPTS: u32 = 3;
62 +
53 63 /// One curated file: where it comes from, what it is, and who to credit.
54 64 #[derive(Debug, Clone, Deserialize)]
55 65 pub struct Asset {
@@ -297,45 +307,119 @@
297 307 }
298 308
299 309 let url = asset.url.as_deref().unwrap_or_default().trim();
310 + let mut last = String::new();
311 + for attempt in 1..=FETCH_ATTEMPTS {
312 + match fetch_once(client, asset, url).await {
313 + Ok(bytes) => {
314 + if let Err(e) = std::fs::write(&cached, &bytes) {
315 + tracing::debug!(id = %asset.id, error = ?e, "example seed: could not cache asset");
316 + }
317 + tracing::info!(id = %asset.id, bytes = bytes.len(), "example seed: fetched media asset");
318 + return Ok(bytes);
319 + }
320 + Err(Fetch::Fatal(reason)) => return Err(reason),
321 + Err(Fetch::Retryable(reason)) => {
322 + tracing::warn!(
323 + id = %asset.id,
324 + attempt,
325 + of = FETCH_ATTEMPTS,
326 + reason = %reason,
327 + "example seed: asset fetch failed; retrying"
328 + );
329 + last = reason;
330 + }
331 + }
332 + }
333 + Err(format!(
334 + "{last} (unchanged over {FETCH_ATTEMPTS} attempts, so this is not a one-off bad response)"
335 + ))
336 + }
337 +
338 + /// Why one attempt failed, and whether another attempt could do better.
339 + enum Fetch {
340 + /// Nothing about trying again would help: the URL is wrong, or the body is
341 + /// over the ceiling.
342 + Fatal(String),
343 + /// The origin may answer differently next time — a transport error, a 5xx or
344 + /// 429, a body that is not the asset, or bytes that miss the pinned digest.
345 + Retryable(String),
346 + }
347 +
348 + /// One HTTP attempt at an asset, verified against the manifest.
349 + async fn fetch_once(client: &reqwest::Client, asset: &Asset, url: &str) -> Result<Vec<u8>, Fetch> {
300 350 let response = client
301 351 .get(url)
302 352 .send()
303 353 .await
304 - .map_err(|e| format!("fetching {url}: {e}"))?;
305 - if !response.status().is_success() {
306 - return Err(format!("fetching {url}: HTTP {}", response.status()));
354 + .map_err(|e| Fetch::Retryable(format!("fetching {url}: {e}")))?;
355 + let status = response.status();
356 + if !status.is_success() {
357 + let reason = format!("fetching {url}: HTTP {status}");
358 + // 5xx and 429 are the origin having a bad moment; a 404 or a 403 is a
359 + // fact about the manifest and retrying only slows the failure down.
360 + return Err(if status.is_server_error() || status.as_u16() == 429 {
361 + Fetch::Retryable(reason)
362 + } else {
363 + Fetch::Fatal(reason)
364 + });
365 + }
366 + // A CDN error page, a challenge, or a consent interstitial is a 200 with a
367 + // non-empty body, and without this it reads as "the asset changed at the
368 + // source" — which sends whoever is holding the red build off to re-check a
369 + // licence that never moved.
370 + //
371 + // The test is deliberately "did we get a DOCUMENT where media was declared",
372 + // not "does the type equal media_type". Content types for the same bytes
373 + // legitimately vary — Wikimedia serves `application/ogg` for files this
374 + // manifest declares `audio/ogg`, which is correct on both ends — so an
375 + // equality check would fail good fetches. Nobody's JPEG is ever text/html.
376 + if let Some(got) = response
377 + .headers()
378 + .get(reqwest::header::CONTENT_TYPE)
379 + .and_then(|v| v.to_str().ok())
380 + .map(|v| v.split(';').next().unwrap_or(v).trim().to_ascii_lowercase())
381 + && is_document(&got)
382 + && !is_document(&asset.media_type.to_ascii_lowercase())
383 + {
384 + return Err(Fetch::Retryable(format!(
385 + "fetching {url}: the origin answered with {got} where the manifest \
386 + declares {}, so this is a page about the asset rather than the \
387 + asset. Its digest says nothing about whether the asset changed.",
388 + asset.media_type
389 + )));
307 390 }
308 391 // Refuse an oversized body before buffering it, when the server declares one.
309 392 if let Some(len) = response.content_length()
310 393 && len > MAX_ASSET_BYTES
311 394 {
312 - return Err(format!(
395 + return Err(Fetch::Fatal(format!(
313 396 "fetching {url}: {len} bytes exceeds the {MAX_ASSET_BYTES}-byte asset ceiling"
314 - ));
397 + )));
315 398 }
316 399 let bytes = response
317 400 .bytes()
318 401 .await
319 - .map_err(|e| format!("reading {url}: {e}"))?
402 + .map_err(|e| Fetch::Retryable(format!("reading {url}: {e}")))?
320 403 .to_vec();
321 404 if bytes.len() as u64 > MAX_ASSET_BYTES {
322 - return Err(format!(
405 + return Err(Fetch::Fatal(format!(
323 406 "fetching {url}: {} bytes exceeds the {MAX_ASSET_BYTES}-byte asset ceiling",
324 407 bytes.len()
325 - ));
408 + )));
326 409 }
327 410 if bytes.is_empty() {
328 - return Err(format!("fetching {url}: empty body"));
411 + return Err(Fetch::Retryable(format!("fetching {url}: empty body")));
329 412 }
330 413
331 414 let got = digest_hex(&bytes);
332 415 match asset.sha256.as_deref().map(str::trim) {
333 416 Some(want) if !want.is_empty() => {
334 417 if !got.eq_ignore_ascii_case(want) {
335 - return Err(format!(
418 + return Err(Fetch::Retryable(format!(
336 419 "digest mismatch for {url}: manifest pins {want}, fetched {got}. \
337 - The asset changed at the source; re-check the licence before repinning."
338 - ));
420 + Either the asset changed at the source (re-check the licence \
421 + before repinning) or this response was not the asset."
422 + )));
339 423 }
340 424 }
341 425 _ => tracing::warn!(
@@ -345,13 +429,24 @@
345 429 ),
346 430 }
347 431
348 - if let Err(e) = std::fs::write(&cached, &bytes) {
349 - tracing::debug!(id = %asset.id, error = ?e, "example seed: could not cache asset");
350 - }
351 - tracing::info!(id = %asset.id, bytes = bytes.len(), "example seed: fetched media asset");
352 432 Ok(bytes)
353 433 }
354 434
435 + /// Whether a lowercased content type names a document rather than a media file.
436 + ///
437 + /// These are the shapes an origin answers with when it is telling you something
438 + /// instead of giving you the bytes: an error page, a bot challenge, a JSON API
439 + /// error. Anything else — including container types like `application/ogg` and
440 + /// the `application/octet-stream` a plain file server falls back to — is
441 + /// treated as media and left to the digest to judge.
442 + fn is_document(content_type: &str) -> bool {
443 + content_type.starts_with("text/")
444 + || matches!(
445 + content_type,
446 + "application/json" | "application/xml" | "application/xhtml+xml"
447 + )
448 + }
449 +
355 450 /// Lowercase hex SHA-256.
356 451 fn digest_hex(bytes: &[u8]) -> String {
357 452 use sha2::{Digest, Sha256};
@@ -367,6 +462,182 @@
367 462 Manifest::parse(EMBEDDED_MANIFEST).expect("the embedded manifest must always be valid");
368 463 }
369 464
465 + /// A curated asset pointed at `url`, pinned to the digest of `body`.
466 + fn pinned_asset(url: &str, body: &[u8]) -> Asset {
467 + Asset {
468 + id: "a".into(),
469 + url: Some(url.into()),
470 + sha256: Some(digest_hex(body)),
471 + media_type: "image/jpeg".into(),
472 + filename: "a.jpg".into(),
473 + license: "CC0-1.0".into(),
474 + title: "A".into(),
475 + author: None,
476 + source: "https://example.test/a".into(),
477 + }
478 + }
479 +
480 + fn seed_client() -> reqwest::Client {
481 + crate::crypto::install_default_crypto_provider();
482 + reqwest::Client::builder().build().unwrap()
483 + }
484 +
485 + #[tokio::test]
486 + async fn a_transient_bad_response_is_retried_rather_than_believed() {
487 + use wiremock::matchers::{method, path};
488 + use wiremock::{Mock, MockServer, ResponseTemplate};
489 +
490 + // Sando build 60 (2026-08-19): seventeen pinned assets came back wrong in
491 + // one burst and every one of them re-fetched byte-identical to its pin
492 + // afterwards. Before this, the first bad response ended the build and the
493 + // message sent whoever read it off to re-check a licence that never moved.
494 + let server = MockServer::start().await;
495 + let good = b"the real asset bytes";
496 + Mock::given(method("GET"))
497 + .and(path("/a.jpg"))
498 + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"a CDN error page".as_ref()))
499 + .up_to_n_times(1)
500 + .mount(&server)
501 + .await;
502 + Mock::given(method("GET"))
503 + .and(path("/a.jpg"))
504 + .respond_with(ResponseTemplate::new(200).set_body_bytes(good.as_ref()))
505 + .mount(&server)
506 + .await;
507 +
508 + let url = format!("{}/a.jpg", server.uri());
509 + let asset = pinned_asset(&url, good);
510 + let cache = tempfile::tempdir().unwrap();
511 +
512 + let bytes = fetch_asset(&seed_client(), cache.path(), &asset)
513 + .await
514 + .expect("the second attempt returns the pinned bytes");
515 + assert_eq!(bytes, good);
516 + }
517 +
518 + #[tokio::test]
519 + async fn a_body_that_is_not_the_declared_type_says_so_instead_of_blaming_the_pin() {
520 + use wiremock::matchers::{method, path};
521 + use wiremock::{Mock, MockServer, ResponseTemplate};
522 +
523 + let server = MockServer::start().await;
524 + Mock::given(method("GET"))
525 + .and(path("/a.jpg"))
526 + .respond_with(
527 + ResponseTemplate::new(200)
528 + .insert_header("content-type", "text/html")
529 + .set_body_bytes(b"<html>are you a robot</html>".as_ref()),
530 + )
531 + .mount(&server)
532 + .await;
533 +
534 + let url = format!("{}/a.jpg", server.uri());
535 + let asset = pinned_asset(&url, b"the real asset bytes");
536 + let cache = tempfile::tempdir().unwrap();
537 +
538 + let err = fetch_asset(&seed_client(), cache.path(), &asset)
539 + .await
540 + .expect_err("an interstitial is not the asset");
541 + assert!(
542 + err.contains("text/html") && err.contains("image/jpeg"),
543 + "the error must name what came back instead: {err}"
544 + );
545 + assert!(
546 + !err.contains("changed at the source"),
547 + "a wrong content type is not evidence the asset changed: {err}"
548 + );
549 + }
550 +
551 + #[tokio::test]
552 + async fn a_container_type_that_differs_from_the_declared_one_is_still_the_asset() {
553 + use wiremock::matchers::{method, path};
554 + use wiremock::{Mock, MockServer, ResponseTemplate};
555 +
556 + // Wikimedia serves `application/ogg` for the three .ogg files this
557 + // manifest declares `audio/ogg`, and both are correct. An equality check
558 + // on content type fails all three, which is how the first cut of the
559 + // document guard was caught.
560 + let server = MockServer::start().await;
561 + let good = b"ogg bytes";
562 + Mock::given(method("GET"))
563 + .and(path("/a.ogg"))
564 + .respond_with(
565 + ResponseTemplate::new(200)
566 + .insert_header("content-type", "application/ogg")
567 + .set_body_bytes(good.as_ref()),
568 + )
569 + .mount(&server)
570 + .await;
571 +
572 + let url = format!("{}/a.ogg", server.uri());
573 + let mut asset = pinned_asset(&url, good);
574 + asset.media_type = "audio/ogg".into();
575 + let cache = tempfile::tempdir().unwrap();
576 +
577 + let bytes = fetch_asset(&seed_client(), cache.path(), &asset)
578 + .await
579 + .expect("application/ogg is an ogg file, not a page about one");
580 + assert_eq!(bytes, good);
581 + }
582 +
583 + #[tokio::test]
584 + async fn a_404_fails_once_rather_than_retrying() {
585 + use wiremock::matchers::{method, path};
586 + use wiremock::{Mock, MockServer, ResponseTemplate};
587 +
588 + // A missing URL is a fact about the manifest. Retrying it only makes the
589 + // build take three times as long to say the same thing.
590 + let server = MockServer::start().await;
591 + Mock::given(method("GET"))
592 + .and(path("/a.jpg"))
593 + .respond_with(ResponseTemplate::new(404))
594 + .expect(1)
595 + .mount(&server)
596 + .await;
597 +
598 + let url = format!("{}/a.jpg", server.uri());
599 + let asset = pinned_asset(&url, b"the real asset bytes");
600 + let cache = tempfile::tempdir().unwrap();
601 +
602 + let err = fetch_asset(&seed_client(), cache.path(), &asset)
603 + .await
604 + .expect_err("a 404 is fatal");
605 + assert!(err.contains("404"), "{err}");
606 + // MockServer asserts the `expect(1)` on drop.
607 + }
608 +
609 + #[tokio::test]
610 + async fn a_digest_that_never_matches_still_fails_after_the_retries() {
611 + use wiremock::matchers::{method, path};
612 + use wiremock::{Mock, MockServer, ResponseTemplate};
613 +
614 + // The retry must not turn a genuine change at the source into a pass.
615 + let server = MockServer::start().await;
616 + Mock::given(method("GET"))
617 + .and(path("/a.jpg"))
618 + .respond_with(
619 + ResponseTemplate::new(200)
620 + .insert_header("content-type", "image/jpeg")
621 + .set_body_bytes(b"different bytes every build".as_ref()),
622 + )
623 + .expect(u64::from(FETCH_ATTEMPTS))
624 + .mount(&server)
625 + .await;
626 +
627 + let url = format!("{}/a.jpg", server.uri());
628 + let asset = pinned_asset(&url, b"the real asset bytes");
629 + let cache = tempfile::tempdir().unwrap();
630 +
631 + let err = fetch_asset(&seed_client(), cache.path(), &asset)
632 + .await
633 + .expect_err("a real change must still fail the seed");
634 + assert!(err.contains("digest mismatch"), "{err}");
635 + assert!(
636 + err.contains("re-check the licence"),
637 + "a persistent mismatch is the case that wants the licence check: {err}"
638 + );
639 + }
640 +
370 641 #[test]
371 642 fn duplicate_ids_are_refused() {
372 643 let text = r#"
M server/src/seed/media-manifest.toml +20
@@ -81,6 +81,26 @@
81 81 # here, the pin is right and the build is what went wrong: rebuild rather than
82 82 # edit this file.
83 83 #
84 + # WHY THE BUILD GOT BYTES fw13 COULD NOT REPRODUCE, settled 2026-08-19. The
85 + # gate had no cache. `SEED_MEDIA_CACHE` defaults under `$TMPDIR`, sandod runs
86 + # with `PrivateTmp=true`, so every Sando build opened on an empty `/tmp` and
87 + # re-downloaded all 34 assets from scratch — while the same fetch from a normal
88 + # shell on the same machine hit a months-old warm cache and never touched the
89 + # network at all. That is the whole discrepancy: not two answers from the Met,
90 + # but one machine asking 34 times per build and the other not asking.
91 + #
92 + # Both halves are now fixed, and neither one edits this file:
93 + #
94 + # - sando.toml points SEED_MEDIA_CACHE at /srv/sando/seed-media-cache, so the
95 + # gate reaches the network only for an asset it does not already hold at the
96 + # pinned digest.
97 + # - manifest.rs retries a mismatch FETCH_ATTEMPTS times and refuses a response
98 + # that is a document rather than media, so a bad minute at a CDN reads as
99 + # what it is instead of as "the asset changed at the source".
100 + #
101 + # All 34 pins were re-verified against the live origins from a cold cache on
102 + # 2026-08-19, after both fixes. Every one matched.
103 + #
84 104 # Two sources, both chosen because the licence is machine-readable at the
85 105 # source rather than asserted by a human reading a page:
86 106 #
M sando/daemon/src/routes/mod.rs +1
@@ -1162,6 +1162,7 @@
1162 1162 AppConfig {
1163 1163 page_smoke_cmd: None,
1164 1164 platform: None,
1165 + code_smoke_env: Default::default(),
1165 1166 id: crate::domain::AppId::default(),
1166 1167 topology_path: PathBuf::from("/tmp/test-sando.toml"),
1167 1168 build_host: Some("test-host".into()),