Skip to main content

max / makenotwork

Land every build's artifacts at one address on astra `pull_root` is per build host and fences what may be pulled off that box; it never said where the result belonged. So a release's bytes ended up in `dist_root` on whichever machine bentod happened to run on, and "where is the AppImage for goingson 1.4.0" had a different answer per target. A new `[archive]` table names the destination once for the fleet. Each target's collect is deposited at `<root>/<app>/<version>/<target>/` on the archive host — astra in production: always on, on the tailnet, and already the aarch64 build host and git mirror, so it is the one box every other build host can reach. Absent table = no archiving. The deposit runs inside `collect`, so a failure fails that step before sign and publish rather than red-marking a release that already shipped. It is a failure and not a warning: a deposit that is quietly skipped leaves the archive path wrong for exactly the release nobody was watching. `dist_root` takes the same per-target layout, which also fixes what it was blocking. Every target shared one `<app>/<version>/`, and collect hashes by listing that directory, so a mac build's artifact record and its TargetOk event claimed the AppImage a sibling had collected minutes earlier. ops-exec gains `SyncOpts.mkpath` (`--mkpath`) plus an `archive_deposit()` preset, since the first build of a version is what makes its directory exist. Off for every other caller, where a missing parent is a typo worth failing on. Two bugs found on the way, unrelated to the above. `expand_tilde_handles_home` set HOME process-wide, which disabled `live_config_smoke` for the whole test binary (it skips when $HOME/.config/bento is absent). With that test live again, its assertions were stale: BB's supernote feature and the supernote-push and everycycle registrations all left bento.toml when those were archived and deregistered.
Author: Max Johnson <me@maxj.phd> · 2026-08-07 01:31 UTC
Signed with PGP, not checked
Commit: 0d3a8c48c196add8aa44dc340890ab0046383128
Parent: eae9061
12 files changed, +546 insertions, -56 deletions
M bento/deploy/README.md +24
@@ -42,6 +42,30 @@
42 42 curl -s http://127.0.0.1:8765/state | python3 -m json.tool
43 43 ```
44 44
45 + ## Where artifacts end up
46 +
47 + Each target's `collect` lands locally at `<dist_root>/<app>/<version>/<target>/`
48 + and is then deposited at `<root>/<app>/<version>/<target>/` on the archive host
49 + named by the `[archive]` table. Same layout both sides, so the two are one tree
50 + at two addresses.
51 +
52 + astra holds the archive: always on, on the tailnet, and already the aarch64 build
53 + host and git mirror, so it is the one box every other build host can reach. That
54 + is what gives "where is the AppImage for goingson 1.4.0" a single answer covering
55 + the macOS and Windows targets too, rather than one per machine that built them.
56 +
57 + The archive host needs the directory to exist and to be writable by the SSH user
58 + bentod reaches it as; everything below it is created per release.
59 +
60 + ```sh
61 + ssh astra 'sudo install -d -o "$USER" -g "$USER" -m 0755 /var/lib/bento/artifacts'
62 + ```
63 +
64 + Retention prunes `dist_root` and `logs_root` on the daemon box and never touches
65 + the archive: the archived copy is the one meant to outlive them. A failed deposit
66 + fails that target's `collect` step, before sign and publish. Leave the `[archive]`
67 + table out to skip archiving entirely.
68 +
45 69 ## Auth (CF2)
46 70
47 71 On a loopback bind, build triggers are reachable only from the daemon's own
M bento/deploy/bento-daemon.toml.example +21 -1
@@ -19,12 +19,32 @@
19 19 # credential files (signing keys, notary creds) relative to here. Never logged.
20 20 secrets_root = "/home/max/Code/_private"
21 21
22 - # Collected artifacts land at <dist_root>/<app>/<version>/.
22 + # Collected artifacts land at <dist_root>/<app>/<version>/<target>/ on this box.
23 23 dist_root = "/home/max/Dist"
24 24
25 25 # Per-step run logs: <logs_root>/<app>/<version>/<target>/<step>.<run_id>.log
26 26 logs_root = "/home/max/.local/state/bento/logs"
27 27
28 + # Where finished artifacts end up for good, whichever host built them. Each
29 + # target's collect is deposited at <root>/<app>/<version>/<target>/ — the same
30 + # layout as dist_root above, so the local copy and the archived one are one tree
31 + # at two addresses. This is what makes "where is the AppImage for goingson 1.4.0"
32 + # have a single answer covering the macOS and Windows targets too.
33 + #
34 + # astra: always on, on the tailnet, already the aarch64 build host and the git
35 + # mirror, so it is the one box every other build host can reliably reach. `local`
36 + # deposits on this machine instead. The root is a path ON THE ARCHIVE HOST and is
37 + # never tilde-expanded.
38 + #
39 + # Leave the whole table out to skip archiving; artifacts then live only in
40 + # dist_root on whichever box runs bentod, and retention eventually prunes them.
41 + # With it set, a failed deposit fails that target's collect step — before sign
42 + # and publish — because a silently-skipped deposit makes the archive path wrong
43 + # for exactly the release nobody was watching.
44 + [archive]
45 + host = "astra"
46 + root = "/var/lib/bento/artifacts"
47 +
28 48 # Optional: override the per-step wall-clock budget (seconds) for EVERY step,
29 49 # replacing the per-kind defaults (build 90m, notarize 60m, sign 15m, ...). A
30 50 # step that runs past its budget fails that step and unwinds the recipe, so a
M bento/daemon/src/artifact_record.rs +24 -23
@@ -22,26 +22,22 @@
22 22 use ops_artifact::{ArtifactRecord, GateRecord, Manifest, Provenance, Scope, Verdict};
23 23 use ops_exec::{Action, LogSink, Step as OpStep};
24 24
25 - /// Where a target's record lands: beside the artifacts, named for the target.
25 + /// Where a target's record lands: beside the artifacts it describes, in that
26 + /// target's own collect directory (`dist_root/<app>/<version>/<target>/`).
26 27 ///
27 - /// One directory holds a whole build (`dist_root/<app>/<version>/`), so the
28 - /// file name carries the target or the second host to finish would overwrite
29 - /// the first's paperwork. `/` is not a path character here, hence the slug.
28 + /// Per target because the manifest is — it names the bytes THIS host produced —
29 + /// and a plain `record.json` because the directory already says which target.
30 30 pub fn record_path(
31 31 dist_root: &std::path::Path,
32 32 app: &AppId,
33 33 version: &Version,
34 34 target: Target,
35 35 ) -> std::path::PathBuf {
36 - dist_root
37 - .join(app.as_str())
38 - .join(version.to_string())
39 - .join(format!("{}.record.json", target_slug(target)))
36 + crate::archive::target_dir(dist_root, app, version, target).join(RECORD_FILE)
40 37 }
41 38
42 - fn target_slug(target: Target) -> String {
43 - target.to_string().replace('/', "-")
44 - }
39 + /// The record's file name inside a target's directory.
40 + pub const RECORD_FILE: &str = "record.json";
45 41
46 42 /// Build the record for a finished target run and write it beside the
47 43 /// artifacts. Never fails a build: logs and returns.
@@ -105,6 +101,18 @@
105 101 app = %ctx.app, target = %ctx.target, digest = %record.digest.short(),
106 102 "wrote artifact record"
107 103 );
104 +
105 + // Re-deposit so the archive holds the paperwork next to the bytes. The
106 + // artifacts themselves went over at `collect`; the record is written after
107 + // the recipe finishes, so it needs this second pass. Non-fatal, like the
108 + // rest of this file: the build is over, and a record that reached the local
109 + // tree but not the archive is worth a log, not a retroactive failure.
110 + if let Some(dir) = path.parent()
111 + && let Err(e) =
112 + crate::archive::deposit(&state.cfg, dir, &ctx.app, &ctx.version, ctx.target).await
113 + {
114 + tracing::error!(app = %ctx.app, target = %ctx.target, error = %e, "could not archive the artifact record");
115 + }
108 116 }
109 117
110 118 /// `rustc --version` on the build host.
@@ -224,9 +232,10 @@
224 232 use super::*;
225 233
226 234 #[test]
227 - fn the_record_is_named_for_its_target_so_siblings_do_not_overwrite() {
228 - // Every target of a build collects into one directory. A fixed file name
229 - // would leave the last host to finish as the only one with paperwork.
235 + fn the_record_sits_in_its_own_targets_directory() {
236 + // Each target collects into its own directory, so the file name is the
237 + // same everywhere and the path is what distinguishes two targets'
238 + // paperwork. A shared directory is what used to make that not true.
230 239 let root = std::path::Path::new("/dist");
231 240 let linux = record_path(
232 241 root,
@@ -243,18 +252,10 @@
243 252 assert_ne!(linux, macos);
244 253 assert_eq!(
245 254 linux,
246 - std::path::Path::new("/dist/goingson/0.4.1/linux-x86_64.record.json")
255 + std::path::Path::new("/dist/goingson/0.4.1/linux-x86_64/record.json")
247 256 );
248 257 }
249 258
250 - #[test]
251 - fn a_target_slug_carries_no_path_separator() {
252 - // It becomes one path component; a `/` would silently make it two.
253 - let slug = target_slug("macos/aarch64".parse().unwrap());
254 - assert!(!slug.contains('/'), "{slug} must be a single component");
255 - assert_eq!(slug, "macos-aarch64");
256 - }
257 -
258 259 #[test]
259 260 fn a_passing_step_reports_its_duration() {
260 261 let (v, s) = verdict_of(
M bento/daemon/src/config.rs +116 -2
@@ -14,8 +14,15 @@
14 14 /// Root of the Syncthing private layer (`~/Code/_private`); the `secret()`
15 15 /// host function reads credential files relative to here. Never logged.
16 16 pub secrets_root: PathBuf,
17 - /// Where collected artifacts land (`<dist_root>/<app>/<version>/`).
17 + /// Where collected artifacts land on the daemon's own box
18 + /// (`<dist_root>/<app>/<version>/<target>/`).
18 19 pub dist_root: PathBuf,
20 + /// Where every build's finished artifacts are deposited, whichever host
21 + /// produced them (see [`Archive`]). Unset = artifacts stay on whichever box
22 + /// bentod happens to run on, which is what "where is the AppImage for
23 + /// goingson 1.4.0" had no single answer to.
24 + #[serde(default)]
25 + pub archive: Option<Archive>,
19 26 /// Root for per-step run logs
20 27 /// (`<logs_root>/<app>/<version>/<target>/<step>.<run_id>.log`).
21 28 #[serde(default = "default_logs_root")]
@@ -56,6 +63,31 @@
56 63 pub deploy_installer: String,
57 64 }
58 65
66 + /// The one place a finished artifact ends up, named once for the whole fleet.
67 + ///
68 + /// `pull_root` is the *collection* root the sync gate fences on each build host;
69 + /// it says what may be pulled off that box, not where the result belongs. So
70 + /// before this, a release's bytes ended up wherever bentod was running, and the
71 + /// answer to "where is the AppImage for goingson 1.4.0" depended on which host
72 + /// built it.
73 + ///
74 + /// Deposited by the daemon after each target's `collect`, so it holds every
75 + /// target of every app — including the macOS and Windows ones bentod does not
76 + /// run on — under one versioned path.
77 + #[derive(Debug, Clone, Deserialize)]
78 + pub struct Archive {
79 + /// SSH destination of the archive host (a tailnet alias like `astra`, or
80 + /// `user@host`); `local` deposits on the daemon's own filesystem.
81 + ///
82 + /// astra in production: always on, on the tailnet, and already the aarch64
83 + /// build host and git mirror, so it is the one box every other build host
84 + /// can reliably reach.
85 + pub host: String,
86 + /// Absolute root ON THE ARCHIVE HOST, e.g. `/var/lib/bento/artifacts`.
87 + /// Never tilde-expanded — it names a location on the far side, not here.
88 + pub root: PathBuf,
89 + }
90 +
59 91 fn default_true() -> bool {
60 92 true
61 93 }
@@ -73,7 +105,34 @@
73 105 let path = std::env::var("BENTO_CONFIG").unwrap_or_else(|_| "bento-daemon.toml".into());
74 106 let raw = std::fs::read_to_string(&path)
75 107 .with_context(|| format!("reading daemon config at {path}"))?;
76 - Ok(toml::from_str(&raw)?)
108 + let cfg: Self = toml::from_str(&raw)?;
109 + cfg.validate()
110 + .with_context(|| format!("daemon config at {path}"))?;
111 + Ok(cfg)
112 + }
113 +
114 + /// Check what would otherwise only fail mid-release. The archive root is
115 + /// interpolated into an rsync destination on another machine, so a relative
116 + /// path or a `..` would deposit a signed release somewhere other than where
117 + /// the config appears to say — the same rule, and the same reason, as
118 + /// `topology::validate_deploy`'s check on `install_path`.
119 + fn validate(&self) -> Result<()> {
120 + if let Some(a) = &self.archive {
121 + anyhow::ensure!(
122 + !a.host.trim().is_empty(),
123 + "[archive] host is empty; set it to an ssh destination or `local`"
124 + );
125 + anyhow::ensure!(
126 + a.root.is_absolute()
127 + && !a
128 + .root
129 + .components()
130 + .any(|c| c == std::path::Component::ParentDir),
131 + "[archive] root `{}` must be an absolute path with no `..`",
132 + a.root.display()
133 + );
134 + }
135 + Ok(())
77 136 }
78 137
79 138 #[cfg(test)]
@@ -84,6 +143,9 @@
84 143 topology_path: root.join("bento.toml"),
85 144 secrets_root: root.join("secrets"),
86 145 dist_root: root.join("dist"),
146 + // Off by default in tests: the archive is a second machine, and the
147 + // tests that exercise it point it at a local directory themselves.
148 + archive: None,
87 149 logs_root: root.join("logs"),
88 150 step_timeout_secs: None,
89 151 notarize_backoff_secs: None,
@@ -94,3 +156,55 @@
94 156 }
95 157 }
96 158 }
159 +
160 + #[cfg(test)]
161 + mod tests {
162 + use super::*;
163 +
164 + /// Parse a whole daemon config with `body` appended, through the same
165 + /// validation `load()` runs.
166 + fn parse(body: &str) -> Result<Config> {
167 + let base = r#"
168 + listen = "127.0.0.1:8765"
169 + db_path = "/var/lib/bento/bento.db"
170 + topology_path = "/etc/bento/bento.toml"
171 + secrets_root = "/home/max/Code/_private"
172 + dist_root = "/home/max/Dist"
173 + "#;
174 + let cfg: Config = toml::from_str(&format!("{base}{body}"))?;
175 + cfg.validate()?;
176 + Ok(cfg)
177 + }
178 +
179 + /// The table is optional: every config written before the archive existed
180 + /// must keep loading, and an operator who has not named an archive host
181 + /// still gets releases.
182 + #[test]
183 + fn the_archive_table_is_optional() {
184 + assert!(parse("").unwrap().archive.is_none());
185 + }
186 +
187 + #[test]
188 + fn an_archive_host_and_root_parse() {
189 + let cfg =
190 + parse("[archive]\nhost = \"astra\"\nroot = \"/var/lib/bento/artifacts\"\n").unwrap();
191 + let a = cfg.archive.expect("archive configured");
192 + assert_eq!(a.host, "astra");
193 + assert_eq!(a.root, PathBuf::from("/var/lib/bento/artifacts"));
194 + }
195 +
196 + /// The root reaches an rsync destination on another machine. A relative path
197 + /// lands in the ssh user's home and a `..` walks out of the declared tree —
198 + /// both put a signed release somewhere other than where the config reads as
199 + /// saying, which is worth failing at startup rather than mid-release.
200 + #[test]
201 + fn a_relative_or_dot_dot_archive_root_is_rejected() {
202 + assert!(parse("[archive]\nhost = \"astra\"\nroot = \"artifacts\"\n").is_err());
203 + assert!(parse("[archive]\nhost = \"astra\"\nroot = \"/var/../etc/bento\"\n").is_err());
204 + }
205 +
206 + #[test]
207 + fn an_empty_archive_host_is_rejected() {
208 + assert!(parse("[archive]\nhost = \"\"\nroot = \"/var/lib/bento/artifacts\"\n").is_err());
209 + }
210 + }
M bento/daemon/src/engine.rs +44 -10
@@ -651,9 +651,7 @@
651 651 /// Every artifact this run collected, `file name -> sha256`.
652 652 ///
653 653 /// Already computed at `collect`, which is the only moment the bytes are
654 - /// known to be the ones that landed. Re-reading the directory afterwards
655 - /// would pick up a sibling target's files too, since the whole build shares
656 - /// one `dist_root/<app>/<version>/`.
654 + /// known to be the ones that landed.
657 655 pub fn artifact_hashes(&self) -> HashMap<String, String> {
658 656 self.artifact_hashes.lock().unwrap().clone()
659 657 }
@@ -1802,8 +1800,23 @@
1802 1800 Ok(tail)
1803 1801 }
1804 1802
1803 + /// Where this run's collected files land locally.
1804 + ///
1805 + /// Per target, not per version. Every target used to share one
1806 + /// `dist_root/<app>/<version>/`, so the hash loop below (which lists the
1807 + /// directory) attributed a sibling's AppImage to the mac build's artifact
1808 + /// record. It is also the layout the archive uses, and the two have to agree
1809 + /// or the local copy and the deposited one are different shapes.
1810 + fn collect_dest(&self, app: &str, version: &str) -> PathBuf {
1811 + self.cfg
1812 + .dist_root
1813 + .join(app)
1814 + .join(version)
1815 + .join(crate::archive::target_slug(self.target))
1816 + }
1817 +
1805 1818 fn collect(self: &Arc<Self>, host: &str, glob: &str, app: &str, version: &str) -> Result<()> {
1806 - let dest = self.cfg.dist_root.join(app).join(version);
1819 + let dest = self.collect_dest(app, version);
1807 1820 let dest_s = dest.to_string_lossy().into_owned();
1808 1821 // The glob reaches a remote login shell intact (that's what expands it),
1809 1822 // so command metacharacters stay barred. Path/wildcard chars are fine.
@@ -1839,6 +1852,25 @@
1839 1852 self.artifact_hashes.lock().unwrap().insert(name, digest);
1840 1853 }
1841 1854 }
1855 + // Deposit at the archive path, so this target's bytes have one address
1856 + // whichever host produced them. A no-op when no archive is configured.
1857 + //
1858 + // Inside `collect`, not after the recipe: a failure here fails the
1859 + // collect step, before sign and publish, rather than putting a red mark
1860 + // on a release that has already shipped. And it is a failure, not a
1861 + // warning — a deposit that is quietly skipped leaves the archive path
1862 + // wrong for exactly the release nobody was watching, which is the thing
1863 + // having one address is for.
1864 + let (cfg, app_id, version, target) = (
1865 + self.cfg.clone(),
1866 + self.app.clone(),
1867 + self.version.clone(),
1868 + self.target,
1869 + );
1870 + let dest_archive = dest.clone();
1871 + self.run_bounded("deposit in the archive", async move {
1872 + crate::archive::deposit(&cfg, &dest_archive, &app_id, &version, target).await
1873 + })?;
1842 1874 // Best-effort size accounting for the event.
1843 1875 events::emit(
1844 1876 &self.events,
@@ -1986,10 +2018,7 @@
1986 2018 if p.is_absolute() {
1987 2019 p
1988 2020 } else {
1989 - self.cfg
1990 - .dist_root
1991 - .join(app.as_str())
1992 - .join(version.to_string())
2021 + self.collect_dest(app.as_str(), &version.to_string())
1993 2022 .join(artifact)
1994 2023 }
1995 2024 };
@@ -2579,10 +2608,15 @@
2579 2608 assert!(m.description.is_none());
2580 2609 }
2581 2610
2611 + /// Reads the ambient `HOME` rather than setting one. `set_var` is
2612 + /// process-global and unsynchronized, so a test that overwrote HOME changed
2613 + /// it for every other test in the binary — which is what silently disabled
2614 + /// `topology::live_config_smoke` (it skips when `$HOME/.config/bento` is
2615 + /// absent, and `/home/test` always is).
2582 2616 #[test]
2583 2617 fn expand_tilde_handles_home() {
2584 - unsafe { std::env::set_var("HOME", "/home/test") };
2585 - assert_eq!(expand_tilde("~/Code/x"), PathBuf::from("/home/test/Code/x"));
2618 + let home = PathBuf::from(std::env::var("HOME").expect("HOME is set"));
2619 + assert_eq!(expand_tilde("~/Code/x"), home.join("Code/x"));
2586 2620 assert_eq!(expand_tilde("/abs/path"), PathBuf::from("/abs/path"));
2587 2621 }
2588 2622
M bento/daemon/src/lib.rs +1
@@ -8,6 +8,7 @@
8 8 //! Design: `bento-overview` in the shared code wiki (`~/Code/_private/wiki/`).
9 9 //! <!-- wiki: bento-overview -->
10 10
11 + pub mod archive;
11 12 pub mod artifact_record;
12 13 pub mod config;
13 14 pub mod db;
M bento/daemon/src/retention.rs +2
@@ -35,6 +35,8 @@
35 35 /// errors are logged and skipped, never fatal. Intended to run inside
36 36 /// `spawn_blocking`.
37 37 pub fn prune_once(cfg: &Config, protected: &ProtectedVersions) {
38 + // The daemon's own copies only. The archive host is deliberately left alone:
39 + // it is the copy meant to outlive the build box's disk pressure.
38 40 prune_root(&cfg.dist_root, "dist", protected);
39 41 prune_root(&cfg.logs_root, "logs", protected);
40 42 }
M bento/daemon/src/runner.rs +40 -11
@@ -657,7 +657,7 @@
657 657
658 658 match outcome {
659 659 Ok(Ok(())) => {
660 - let artifacts = collected_artifacts(&state, &app, &version);
660 + let artifacts = collected_artifacts(&state, &app, &version, target);
661 661 if let Err(e) = sqlx::query(
662 662 "UPDATE target_runs SET status = 'ok', current_step = NULL, finished_at = ? WHERE id = ?",
663 663 )
@@ -896,12 +896,18 @@
896 896 std::fs::read_to_string(&path).with_context(|| format!("reading recipe {}", path.display()))
897 897 }
898 898
899 - fn collected_artifacts(state: &AppState, app: &AppId, version: &Version) -> Vec<String> {
900 - let dir = state
901 - .cfg
902 - .dist_root
903 - .join(app.as_str())
904 - .join(version.to_string());
899 + /// What this target run left in its collect directory.
900 + ///
901 + /// Per target: the event reports what THIS run produced, and every target of a
902 + /// version used to share one directory, so a mac build's `TargetOk` listed the
903 + /// Linux AppImage a sibling had collected minutes earlier.
904 + fn collected_artifacts(
905 + state: &AppState,
906 + app: &AppId,
907 + version: &Version,
908 + target: Target,
909 + ) -> Vec<String> {
910 + let dir = crate::archive::target_dir(&state.cfg.dist_root, app, version, target);
905 911 let Ok(rd) = std::fs::read_dir(&dir) else {
906 912 return Vec::new();
907 913 };
@@ -1326,7 +1332,13 @@
1326 1332 )
1327 1333 .unwrap();
1328 1334
1329 - let cfg = Config::for_tests(root);
1335 + let mut cfg = Config::for_tests(root);
1336 + // Archive to a local directory, so the deposit a real release makes to
1337 + // astra runs on the same code path here.
1338 + cfg.archive = Some(crate::config::Archive {
1339 + host: "local".into(),
1340 + root: root.join("archive"),
1341 + });
1330 1342 let pool = crate::db::open(&cfg.db_path).await.unwrap();
1331 1343 std::fs::write(repo.join("bento.toml"), "targets = [\"linux/x86_64\"]\n").unwrap();
1332 1344 let topo = Topology::from_str_for_tests(&format!(
@@ -1386,9 +1398,22 @@
1386 1398 assert_eq!(names, vec!["build", "collect"]);
1387 1399 assert!(steps.iter().all(|(_, st)| st == "ok"));
1388 1400
1389 - // Artifact landed in dist_root and a step log was written.
1390 - let artifact = state.cfg.dist_root.join("demo/0.0.1/demo.bin");
1401 + // Artifact landed in dist_root, under its own target, and a step log was
1402 + // written. Both trees are keyed the same way, `<app>/<version>/<target>/`.
1403 + let artifact = state.cfg.dist_root.join("demo/0.0.1/linux-x86_64/demo.bin");
1391 1404 assert!(artifact.exists(), "collect should copy the artifact");
1405 + // ...and the same bytes reached the archive, at the same path under its
1406 + // own root. This is the answer to "where is demo 0.0.1 for linux".
1407 + let archived = root.join("archive/demo/0.0.1/linux-x86_64/demo.bin");
1408 + assert!(
1409 + archived.exists(),
1410 + "collect should deposit into the archive: {}",
1411 + archived.display()
1412 + );
1413 + assert_eq!(
1414 + std::fs::read(&archived).unwrap(),
1415 + std::fs::read(&artifact).unwrap()
1416 + );
1392 1417 // Read the path off the ledger rather than rebuilding it: the log is
1393 1418 // named for its step run id, and the point of that is that the row
1394 1419 // resolves to exactly one file.
@@ -1541,7 +1566,11 @@
1541 1566
1542 1567 // The surviving target finished its work, not merely its row.
1543 1568 assert!(
1544 - state.cfg.dist_root.join("demo/0.0.1/demo.bin").exists(),
1569 + state
1570 + .cfg
1571 + .dist_root
1572 + .join("demo/0.0.1/linux-x86_64/demo.bin")
1573 + .exists(),
1545 1574 "the succeeding target ran to completion and collected its artifact",
1546 1575 );
1547 1576
M bento/daemon/src/topology.rs +5 -9
@@ -808,14 +808,8 @@
808 808 return;
809 809 }
810 810 let topo = Topology::load(&path).expect("live bento.toml must load");
811 - let bb = topo
812 - .app(&"balanced_breakfast".into())
811 + topo.app(&"balanced_breakfast".into())
813 812 .expect("bb configured");
814 - assert!(
815 - bb.features.contains(&"supernote".to_string()),
816 - "bb must ship the supernote feature, got {:?}",
817 - bb.features
818 - );
819 813 let af = topo
820 814 .app(&"audiofiles".into())
821 815 .expect("audiofiles configured");
@@ -825,8 +819,10 @@
825 819 );
826 820
827 821 // The library crates resolve as libraries, so they take publish.rhai
828 - // rather than a per-platform recipe.
829 - for name in ["makeover", "pter", "everycycle", "supernote-push"] {
822 + // rather than a per-platform recipe. The whole makeover suite is here;
823 + // this names the ends of it plus one crate outside it, since the point
824 + // is the `kind` resolution and not a roll call of the registry.
825 + for name in ["makeover", "makeover-touch", "pter", "alloy_tui"] {
830 826 let c = topo
831 827 .app(&name.into())
832 828 .unwrap_or_else(|| panic!("{name} configured"));
M shared/ops-exec/src/executor.rs +19
@@ -34,6 +34,13 @@
34 34 /// merely useless — e.g. fetching a DB dump, where a resumed transfer could
35 35 /// splice two different dumps into one plausible-looking file.
36 36 pub partial: bool,
37 + /// `--mkpath`: create the destination's missing parent directories rather
38 + /// than failing. Off by default, because for a hand-written destination a
39 + /// missing parent is a typo and minting the tree hides it. Turn it on when
40 + /// the caller computes the destination and is authoritative for it — Bento's
41 + /// per-`(app, version, target)` artifact directory, which by definition does
42 + /// not exist before the build that fills it.
43 + pub mkpath: bool,
37 44 }
38 45
39 46 impl Default for SyncOpts {
@@ -43,6 +50,7 @@
43 50 chmod: None,
44 51 compress: true,
45 52 partial: true,
53 + mkpath: false,
46 54 }
47 55 }
48 56 }
@@ -66,6 +74,17 @@
66 74 ..Self::default()
67 75 }
68 76 }
77 +
78 + /// Depositing an already-compressed release artifact at a computed,
79 + /// versioned destination: no `-z`, and create the destination tree, since
80 + /// the first build of a version is the thing that makes its directory exist.
81 + pub fn archive_deposit() -> Self {
82 + Self {
83 + compress: false,
84 + mkpath: true,
85 + ..Self::default()
86 + }
87 + }
69 88 }
70 89
71 90 /// A read-only host observation. v1 can synthesize these from SSH-streamed
M shared/ops-exec/src/transport.rs +34
@@ -160,6 +160,9 @@
160 160 if let Some(chmod) = &opts.chmod {
161 161 rsync.arg(format!("--chmod={chmod}"));
162 162 }
163 + if opts.mkpath {
164 + rsync.arg("--mkpath");
165 + }
163 166 if let Some(args) = ssh_args {
164 167 rsync.arg("-e").arg(format!("ssh {}", args.join(" ")));
165 168 }
@@ -778,6 +781,37 @@
778 781 assert!(args.contains(&"--delete".to_string()));
779 782 assert!(args.iter().any(|a| a.starts_with("--chmod=")));
780 783 assert!(args.contains(&"-z".to_string()));
784 +
785 + // An archive deposit creates its destination tree (the first build of a
786 + // version is what makes that directory exist) and never prunes it: a
787 + // second target of the same release deposits beside the first.
788 + let args = render_args(&rsync_command("s", "d", None, &SyncOpts::archive_deposit()));
789 + assert!(args.contains(&"--mkpath".to_string()), "{args:?}");
790 + assert!(!args.contains(&"--delete".to_string()), "{args:?}");
791 + assert!(!args.contains(&"-z".to_string()), "{args:?}");
792 +
793 + // ...and nothing else creates directories implicitly: a typo'd
794 + // destination for every other caller stays an error.
795 + let args = render_args(&rsync_command("s", "d", None, &SyncOpts::default()));
796 + assert!(!args.contains(&"--mkpath".to_string()), "{args:?}");
797 + }
798 +
799 + /// `--mkpath` end to end: a push into a destination whose parents do not
800 + /// exist creates them, which is the whole reason Bento's archive can name a
801 + /// per-`(app, version, target)` path before that version has ever built.
802 + #[tokio::test]
803 + async fn a_deposit_creates_its_missing_destination_tree() {
804 + let dir = tempfile::tempdir().unwrap();
805 + let src = dir.path().join("src");
806 + std::fs::create_dir_all(&src).unwrap();
807 + std::fs::write(src.join("demo.bin"), b"bytes").unwrap();
808 +
809 + let dst = dir.path().join("archive/demo/0.0.1/linux-x86_64");
810 + let exec = LocalExec::new(CapabilitySet::from_tokens::<[&str; 0], [&str; 0]>([], []));
811 + exec.push_dir(&src, &dst, &SyncOpts::archive_deposit())
812 + .await
813 + .unwrap();
814 + assert_eq!(std::fs::read(dst.join("demo.bin")).unwrap(), b"bytes");
781 815 }
782 816
783 817 #[test]
A bento/daemon/src/archive.rs +216
@@ -1,0 +1,216 @@
1 + //! Depositing finished artifacts at one address, whichever host built them.
2 + //!
3 + //! Design + rationale: maintainer wiki.
4 + //! <!-- wiki: bento-overview -->
5 + //!
6 + //! Bento's `pull_root` is per build host and fences what may be pulled *off*
7 + //! that box. Nothing said where the result belonged, so a release's bytes ended
8 + //! up in `dist_root` on whichever machine bentod happened to run on, and "where
9 + //! is the AppImage for goingson 1.4.0" had a different answer per target.
10 + //!
11 + //! This is the other half: after a target's `collect` lands its files locally,
12 + //! the daemon pushes that directory to `<root>/<app>/<version>/<target>/` on the
13 + //! archive host. The layout matches `dist_root`'s exactly, so the local copy and
14 + //! the archived one are the same tree at two addresses rather than two shapes.
15 + //!
16 + //! Unconfigured, none of this runs. Configured, a failed deposit fails the
17 + //! `collect` step: the point of the archive is that the path IS the answer to
18 + //! where a version's artifact is, and a deposit that is quietly skipped makes
19 + //! that answer wrong for exactly the release nobody watched.
20 +
21 + use crate::config::{Archive, Config};
22 + use crate::domain::{AppId, Target, Version};
23 + use ops_exec::{CapabilitySet, Executor, LocalExec, SshExec, SyncOpts};
24 + use std::path::{Path, PathBuf};
25 + use std::sync::Arc;
26 +
27 + /// One target's directory under a root: `<root>/<app>/<version>/<target>/`.
28 + ///
29 + /// Shared by `dist_root` (locally) and the archive (remotely) so the two trees
30 + /// are identical, and a human who knows one path knows the other.
31 + ///
32 + /// The target is a slug because `/` is a path separator here and
33 + /// `macos/aarch64` would silently become two components.
34 + pub fn target_dir(root: &Path, app: &AppId, version: &Version, target: Target) -> PathBuf {
35 + root.join(app.as_str())
36 + .join(version.to_string())
37 + .join(target_slug(target))
38 + }
39 +
40 + /// `macos/aarch64` -> `macos-aarch64`: one path component, not two.
41 + pub fn target_slug(target: Target) -> String {
42 + target.to_string().replace('/', "-")
43 + }
44 +
45 + /// The transport that writes to the archive host.
46 + ///
47 + /// Granted nothing. Every capability in `ops_exec` gates either running a step
48 + /// or pulling a file, and this executor does neither — it only ever calls
49 + /// `push_dir`. So the archive host cannot be turned into a build host by
50 + /// anything holding this handle, and it is not in the topology, so no recipe can
51 + /// name it. Same shape as `state::build_deploy_executor`, one grant narrower.
52 + fn transport(archive: &Archive) -> Arc<dyn Executor> {
53 + const NONE: [&str; 0] = [];
54 + let caps = CapabilitySet::from_tokens(NONE, NONE);
55 + if archive.host == "local" || archive.host.is_empty() {
56 + return Arc::new(LocalExec::new(caps));
57 + }
58 + Arc::new(SshExec::new(archive.host.clone(), caps))
59 + }
60 +
61 + /// Deposit `local_dir` (a target's collected files) at this target's archive
62 + /// path. A no-op when no archive is configured.
63 + ///
64 + /// Idempotent: rsync without `--delete`, so a re-collect or a retried build
65 + /// re-deposits the same bytes rather than emptying the directory first. Nothing
66 + /// is pruned there either — retention runs against `dist_root` and `logs_root`
67 + /// on the daemon box, and the archive is the copy that is meant to outlive them.
68 + pub async fn deposit(
69 + cfg: &Config,
70 + local_dir: &Path,
71 + app: &AppId,
72 + version: &Version,
73 + target: Target,
74 + ) -> anyhow::Result<()> {
75 + let Some(archive) = &cfg.archive else {
76 + return Ok(());
77 + };
78 + let dest = target_dir(&archive.root, app, version, target);
79 + transport(archive)
80 + .push_dir(local_dir, &dest, &SyncOpts::archive_deposit())
81 + .await
82 + .map_err(|e| {
83 + anyhow::anyhow!(
84 + "depositing {} at {}:{}: {e}",
85 + local_dir.display(),
86 + archive.host,
87 + dest.display()
88 + )
89 + })?;
90 + tracing::info!(
91 + %app, %target, %version, host = %archive.host, dest = %dest.display(),
92 + "deposited artifacts in the archive"
93 + );
94 + Ok(())
95 + }
96 +
97 + #[cfg(test)]
98 + mod tests {
99 + use super::*;
100 +
101 + fn app() -> AppId {
102 + AppId::new("goingson")
103 + }
104 + fn version() -> Version {
105 + "1.4.0".parse().unwrap()
106 + }
107 +
108 + #[test]
109 + fn the_archive_path_is_per_app_version_and_target() {
110 + let dir = target_dir(
111 + Path::new("/var/lib/bento/artifacts"),
112 + &app(),
113 + &version(),
114 + "macos/aarch64".parse().unwrap(),
115 + );
116 + assert_eq!(
117 + dir,
118 + Path::new("/var/lib/bento/artifacts/goingson/1.4.0/macos-aarch64")
119 + );
120 + }
121 +
122 + /// The target has to be one component. A `/` left in would put the aarch64
123 + /// build under a `macos` directory shared with every other mac target, which
124 + /// is the collision the slug exists to prevent.
125 + #[test]
126 + fn a_target_slug_carries_no_path_separator() {
127 + let slug = target_slug("macos/aarch64".parse().unwrap());
128 + assert!(!slug.contains('/'), "{slug} must be a single component");
129 + assert_eq!(slug, "macos-aarch64");
130 + }
131 +
132 + /// Two targets of one version are siblings, not overwrites. This is the
133 + /// property that lets the archive hold a whole release rather than whichever
134 + /// host finished last.
135 + #[test]
136 + fn sibling_targets_do_not_share_a_directory() {
137 + let root = Path::new("/a");
138 + let linux = target_dir(root, &app(), &version(), "linux/x86_64".parse().unwrap());
139 + let macos = target_dir(root, &app(), &version(), "macos/aarch64".parse().unwrap());
140 + assert_ne!(linux, macos);
141 + assert_eq!(linux.parent(), macos.parent());
142 + }
143 +
144 + /// Unconfigured is a no-op rather than an error: an operator who has not
145 + /// named an archive host still gets releases, they just stay local.
146 + #[tokio::test]
147 + async fn no_archive_configured_deposits_nothing() {
148 + let tmp = tempfile::tempdir().unwrap();
149 + let cfg = Config::for_tests(tmp.path());
150 + assert!(cfg.archive.is_none());
151 + deposit(
152 + &cfg,
153 + tmp.path(),
154 + &app(),
155 + &version(),
156 + "linux/x86_64".parse().unwrap(),
157 + )
158 + .await
159 + .expect("a no-op cannot fail");
160 + }
161 +
162 + /// The whole path, against a `local` archive host: files land at
163 + /// `<root>/<app>/<version>/<target>/`, and the destination tree is created
164 + /// (the first build of a version is what makes its directory exist).
165 + #[tokio::test]
166 + async fn a_local_deposit_lands_at_the_versioned_path() {
167 + let tmp = tempfile::tempdir().unwrap();
168 + let src = tmp.path().join("collected");
169 + std::fs::create_dir_all(&src).unwrap();
170 + std::fs::write(src.join("goingson_1.4.0.AppImage"), b"bytes").unwrap();
171 +
172 + let mut cfg = Config::for_tests(tmp.path());
173 + cfg.archive = Some(Archive {
174 + host: "local".into(),
175 + root: tmp.path().join("archive"),
176 + });
177 + let target: Target = "linux/x86_64".parse().unwrap();
178 + deposit(&cfg, &src, &app(), &version(), target)
179 + .await
180 + .unwrap();
181 +
182 + let landed = tmp
183 + .path()
184 + .join("archive/goingson/1.4.0/linux-x86_64/goingson_1.4.0.AppImage");
185 + assert!(landed.exists(), "{} must exist", landed.display());
186 + assert_eq!(std::fs::read(&landed).unwrap(), b"bytes");
187 + }
188 +
189 + /// A second target of the same version deposits beside the first rather than
190 + /// replacing it — the deposit does not `--delete`.
191 + #[tokio::test]
192 + async fn a_second_target_leaves_the_first_alone() {
193 + let tmp = tempfile::tempdir().unwrap();
194 + let mut cfg = Config::for_tests(tmp.path());
195 + cfg.archive = Some(Archive {
196 + host: "local".into(),
197 + root: tmp.path().join("archive"),
198 + });
199 +
200 + for (target, file) in [
201 + ("linux/x86_64", "goingson_1.4.0.AppImage"),
202 + ("macos/aarch64", "goingson_1.4.0.dmg"),
203 + ] {
204 + let src = tmp.path().join(format!("collected-{file}"));
205 + std::fs::create_dir_all(&src).unwrap();
206 + std::fs::write(src.join(file), b"bytes").unwrap();
207 + deposit(&cfg, &src, &app(), &version(), target.parse().unwrap())
208 + .await
209 + .unwrap();
210 + }
211 +
212 + let root = tmp.path().join("archive/goingson/1.4.0");
213 + assert!(root.join("linux-x86_64/goingson_1.4.0.AppImage").exists());
214 + assert!(root.join("macos-aarch64/goingson_1.4.0.dmg").exists());
215 + }
216 + }