Skip to main content

max / makenotwork

Walk a collected bundle the way the verifier will read it Bento's `collect` hashed the top level of the collect directory and keyed by file name; Sando's `bundle::digest_dir` recurses and keys by relative path. For a flat bundle the two agree, which is every bundle anything ships today, so this has never fired. The moment one carries a subdirectory they diverge: bento's manifest omits everything nested, sando re-hashes the bytes that arrived, finds files the manifest never named, and refuses the artifact. An honest release, turned away for corruption, with the offending file named as if something had gone wrong in transit. This is the failure the intake was built to avoid one layer down — it already verifies with Sando's own walker rather than the contract crate's builder, precisely so the prover and the verifier cannot disagree. The producer was the half nobody had matched. `collect` now walks recursively and keys by `/`-separated relative path, sorted, skipping symlinks — the same three properties `digest_dir` has, and each one for a reason: a bundle may carry migrations or resources, following a link would let bytes from outside the bundle into its identity, and readdir order is not guaranteed even on one machine. Safe for `publish`, which looks a hash up by the primary artifact's bare file name: a top-level file's relative path IS its file name, so those keys are unchanged. Nested entries get path keys that only the manifest reads. The version assertion stays on the file name, since what it hunts is a stale `app_1.2.3.AppImage` sitting beside this release's, and a directory component is not that. Pinned from both ends. The same three-file fixture and the same expected manifest text are now a constant in bento's `engine.rs` and in sando's `bundle.rs`. If either walk drifts, that crate's own test fails and names the drift, instead of a release failing intake for a bundle nothing is wrong with. The fixture is deliberately nested, because flat is the case that was already working.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-07 02:57 UTC
Signed with PGP, not checked
Commit: b0d8e6fdf58f38eed05a35352619df492997dbfb
Parent: 16f104f
2 files changed, +174 insertions, -9 deletions
M bento/daemon/src/engine.rs +140 -9
@@ -969,6 +969,53 @@
969 969 Ok(hex_lower(&hasher.finalize()))
970 970 }
971 971
972 + /// Every regular file under `root`, as `(path relative to root, absolute path)`,
973 + /// sorted by the relative path.
974 + ///
975 + /// **This walk has to match `bundle::digest_dir` in sando, file for file.** That
976 + /// function re-hashes an incoming bundle and refuses it when the bytes disagree
977 + /// with the manifest they arrived with, so a producer that walks differently
978 + /// produces a manifest the consumer will reject for an artifact nothing is wrong
979 + /// with. Three properties carry that agreement, and none is incidental:
980 + ///
981 + /// - **Recursive.** A bundle may carry a directory (migrations, resources), and
982 + /// a top-level-only listing would omit its contents from the manifest while
983 + /// the verifier hashed them.
984 + /// - **Symlinks are not followed, and not recorded.** Following one would let
985 + /// content from outside the bundle into its identity; recording the link
986 + /// itself would name a file the verifier does not hash.
987 + /// - **Relative paths, `/`-separated, sorted.** Readdir order is not guaranteed,
988 + /// so an unsorted manifest would differ run to run on one machine, never mind
989 + /// between two.
990 + fn collected_files(root: &Path) -> std::io::Result<Vec<(String, PathBuf)>> {
991 + fn walk(dir: &Path, root: &Path, out: &mut Vec<(String, PathBuf)>) -> std::io::Result<()> {
992 + for entry in std::fs::read_dir(dir)? {
993 + let entry = entry?;
994 + let ft = entry.file_type()?;
995 + let path = entry.path();
996 + if ft.is_dir() {
997 + walk(&path, root, out)?;
998 + } else if ft.is_file() {
999 + let rel = path
1000 + .strip_prefix(root)
1001 + .unwrap_or(&path)
1002 + .components()
1003 + .map(|c| c.as_os_str().to_string_lossy())
1004 + .collect::<Vec<_>>()
1005 + .join("/");
1006 + out.push((rel, path));
1007 + }
1008 + // Symlinks and other special files are intentionally ignored,
1009 + // matching the verifier.
1010 + }
1011 + Ok(())
1012 + }
1013 + let mut out = Vec::new();
1014 + walk(root, root, &mut out)?;
1015 + out.sort_by(|a, b| a.0.cmp(&b.0));
1016 + Ok(out)
1017 + }
1018 +
972 1019 /// Lowercase-hex encode without pulling in a hex crate.
973 1020 fn hex_lower(bytes: &[u8]) -> String {
974 1021 use std::fmt::Write as _;
@@ -1847,17 +1894,29 @@
1847 1894 // Assert the version and hash every collected file. This is where a
1848 1895 // stale artifact is caught: a file whose name embeds a different version
1849 1896 // fails the collect (rather than silently winning a later glob), and the
1850 - // sha256 recorded here is what `publish` writes into the release ledger.
1851 - for entry in
1852 - std::fs::read_dir(&dest).with_context(|| format!("listing collect dest {dest_s}"))?
1897 + // sha256 recorded here is what `publish` writes into the release ledger
1898 + // and what the artifact record's manifest is built from.
1899 + //
1900 + // Recursive, and keyed by path relative to the collect dir. That is not
1901 + // a preference: Sando's intake re-hashes the bundle with its own walker,
1902 + // which recurses and keys the same way, and refuses a bundle whose bytes
1903 + // do not match the manifest it was handed. A top-level `read_dir` keyed
1904 + // by file name agrees with that walker for a flat directory and diverges
1905 + // the moment a bundle carries a subdirectory — the honest artifact would
1906 + // be refused for a manifest that omitted everything nested. The two
1907 + // walkers have to be the same walk. See `bundle::digest_dir` in sando.
1908 + for (rel, path) in
1909 + collected_files(&dest).with_context(|| format!("listing collect dest {dest_s}"))?
1853 1910 {
1854 - let entry = entry?;
1855 - let name = entry.file_name().to_string_lossy().into_owned();
1911 + // The version check stays on the file NAME rather than the relative
1912 + // path: it is looking for a stale `app_1.2.3.AppImage` beside the
1913 + // one this release built, and a directory component is not that.
1914 + let name = path
1915 + .file_name()
1916 + .map_or_else(|| rel.clone(), |n| n.to_string_lossy().into_owned());
1856 1917 assert_artifact_version(&name, &self.version)?;
1857 - if entry.file_type().is_ok_and(|t| t.is_file()) {
1858 - let digest = sha256_file(&entry.path())?;
1859 - self.artifact_hashes.lock().unwrap().insert(name, digest);
1860 - }
1918 + let digest = sha256_file(&path)?;
1919 + self.artifact_hashes.lock().unwrap().insert(rel, digest);
1861 1920 }
1862 1921 // Deposit at the archive path, so this target's bytes have one address
1863 1922 // whichever host produced them. A no-op when no archive is configured.
@@ -2295,6 +2354,78 @@
2295 2354 mod tests {
2296 2355 use super::*;
2297 2356
2357 + /// Build the shared cross-crate bundle fixture under `root`.
2358 + ///
2359 + /// A binary at the top and two files in a subdirectory — the shape a service
2360 + /// that ships its migrations has, which is the case the flat walk used to
2361 + /// get wrong.
2362 + pub(crate) fn write_bundle_fixture(root: &Path) {
2363 + std::fs::create_dir_all(root.join("migrations")).unwrap();
2364 + std::fs::write(root.join("pom"), b"binary-bytes").unwrap();
2365 + std::fs::write(root.join("migrations/001_init.sql"), b"create table a;").unwrap();
2366 + std::fs::write(root.join("migrations/002_next.sql"), b"alter table a;").unwrap();
2367 + }
2368 +
2369 + /// The manifest text the fixture must produce, in BOTH crates.
2370 + ///
2371 + /// Sando's `bundle::digest_dir` has the identical constant and the identical
2372 + /// fixture. That is the whole point: bento writes this text into the artifact
2373 + /// record, sando recomputes it from the bytes that arrive, and an artifact is
2374 + /// refused when they differ. Two walks, one answer, pinned from both ends —
2375 + /// if either crate's walk drifts, its own test fails and names the drift
2376 + /// rather than a release failing intake for a bundle nothing is wrong with.
2377 + pub(crate) const BUNDLE_FIXTURE_MANIFEST: &str = concat!(
2378 + "e4c908e219c533fa7ad7ea1634398f9bf51637ba20717769ada545bab26d7368 migrations/001_init.sql\n",
2379 + "b026fd51bae096b34672cefdb781b6585b13efb53bc301d50c305f422552a380 migrations/002_next.sql\n",
2380 + "71227a7f160afca3fb3c39f448735886dda7bd366252580c2222fb87d4bb4d85 pom\n",
2381 + );
2382 +
2383 + /// The producer half of the contract above: what `collect` hashes, turned
2384 + /// into a manifest, is exactly the text the verifier will recompute.
2385 + ///
2386 + /// Nested files are included and addressed by relative path. Before this,
2387 + /// `collect` listed only the top level, so `migrations/` contributed nothing
2388 + /// to the manifest while sando's walker hashed both files in it — and the
2389 + /// honest bundle was refused for a manifest that had omitted them.
2390 + #[test]
2391 + fn a_collected_bundle_manifests_exactly_as_the_verifier_will_read_it() {
2392 + let dir = tempfile::tempdir().unwrap();
2393 + write_bundle_fixture(dir.path());
2394 +
2395 + let files = collected_files(dir.path()).unwrap();
2396 + assert_eq!(
2397 + files.iter().map(|(r, _)| r.as_str()).collect::<Vec<_>>(),
2398 + vec!["migrations/001_init.sql", "migrations/002_next.sql", "pom"],
2399 + "recursive, relative, sorted"
2400 + );
2401 +
2402 + let hashes: Vec<(String, String)> = files
2403 + .into_iter()
2404 + .map(|(rel, path)| (rel, sha256_file(&path).unwrap()))
2405 + .collect();
2406 + let manifest = ops_artifact::Manifest::new(hashes).unwrap();
2407 + assert_eq!(manifest.to_text(), BUNDLE_FIXTURE_MANIFEST);
2408 + }
2409 +
2410 + /// A symlink is neither followed nor named. Following one would let bytes
2411 + /// from outside the bundle into its identity; naming it would put a path in
2412 + /// the manifest the verifier does not hash, which reads as a corrupt bundle.
2413 + #[test]
2414 + #[cfg(unix)]
2415 + fn a_symlink_in_the_collect_dir_is_not_part_of_the_bundle() {
2416 + let dir = tempfile::tempdir().unwrap();
2417 + write_bundle_fixture(dir.path());
2418 + let outside = dir.path().join("..").join("secret.env");
2419 + std::fs::write(&outside, b"TOKEN=1").ok();
2420 + std::os::unix::fs::symlink(&outside, dir.path().join("link.env")).unwrap();
2421 +
2422 + let files = collected_files(dir.path()).unwrap();
2423 + assert!(
2424 + !files.iter().any(|(rel, _)| rel.contains("link.env")),
2425 + "{files:?}"
2426 + );
2427 + }
2428 +
2298 2429 /// Run 3 S1: once the cooperative cancel flag is set (a newer build
2299 2430 /// superseded this run), a step boundary refuses to proceed — the blocking
2300 2431 /// recipe stops at the next `step()` instead of running on and publishing.
M sando/daemon/src/bundle.rs +34
@@ -150,6 +150,40 @@
150 150 mod tests {
151 151 use super::*;
152 152
153 + /// The manifest text the shared cross-crate fixture must produce, in BOTH
154 + /// crates.
155 + ///
156 + /// Bento's `engine.rs` has the identical constant and the identical fixture.
157 + /// Bento writes this text into the artifact record at `collect`; this walker
158 + /// recomputes it from the bytes that arrive at intake, and an artifact whose
159 + /// two answers differ is refused. Producer and consumer are different walks
160 + /// in different repos, so the agreement is pinned from both ends — if either
161 + /// drifts, its own test fails and names the drift, instead of a release
162 + /// failing intake for a bundle nothing is wrong with.
163 + ///
164 + /// The nested directory is the case that matters: bento's collect used to
165 + /// list only the top level, so `migrations/` contributed nothing to the
166 + /// manifest while this walker hashed both files in it.
167 + const BUNDLE_FIXTURE_MANIFEST: &str = concat!(
168 + "e4c908e219c533fa7ad7ea1634398f9bf51637ba20717769ada545bab26d7368 migrations/001_init.sql\n",
169 + "b026fd51bae096b34672cefdb781b6585b13efb53bc301d50c305f422552a380 migrations/002_next.sql\n",
170 + "71227a7f160afca3fb3c39f448735886dda7bd366252580c2222fb87d4bb4d85 pom\n",
171 + );
172 +
173 + /// The consumer half of the contract: what arrives is read exactly the way
174 + /// the producer described it.
175 + #[tokio::test]
176 + async fn the_shared_bundle_fixture_digests_as_the_producer_wrote_it() {
177 + let tmp = tempfile::tempdir().unwrap();
178 + let root = tmp.path();
179 + write(root, "pom", b"binary-bytes").await;
180 + write(root, "migrations/001_init.sql", b"create table a;").await;
181 + write(root, "migrations/002_next.sql", b"alter table a;").await;
182 +
183 + let d = digest_dir(root).await.unwrap();
184 + assert_eq!(d.manifest, BUNDLE_FIXTURE_MANIFEST);
185 + }
186 +
153 187 async fn write(root: &Path, rel: &str, bytes: &[u8]) {
154 188 let p = root.join(rel);
155 189 tokio::fs::create_dir_all(p.parent().unwrap())