Skip to main content

max / makenotwork

Ship multithreaded as a Sando companion mt had no sanctioned deploy path. deploy-hetzner.sh cross-compiled on macOS via cargo zigbuild and scp'd to root@, against the native-build rule, and it was 41 commits behind when this was found. mt now rides the server's pipeline: native build on fw13, the migration dry run against its own dump, cargo_test, a fatal frontend build, burn-in on testnot, and a binary shipped from the server's sha. Embed static/ in the binary. A companion is exactly one installed file, and build.rs bakes a content hash of style.css, mt.js and static/dist into the templates compiled into the binary, so installing the binary alone would serve new markup against old CSS with nothing reporting an error. include_dir over static/ makes the two inseparable. The ETag matters: the app-wide layer sets private, no-cache, so without a 304 path every navigation re-pulls 1.9M of fonts that ServeDir answered via Last-Modified. Cross-check node companions against built ones. Companions install after the symlink swap, so a node naming a companion the daemon never builds fails a promote with the server already live on prod. Now it fails --check-config on the build host instead. Validate the daemon config we actually ship. The guard above turned check_config_accepts_the_shipped_topology red, because that test hand-rolled a minimal config rather than reading deploy/sando-daemon.toml.example, which left the production daemon config validated by nothing. It was missing both the companion blocks and the test_target list — on the documented prod config the single-server default applies, so mt would have shipped to prod untested. bin_names is not where a companion goes: that list is the server crate's bin targets, resolved under its own target/release. The wiki sketch had it wrong. Host setup is still owed on testnot-1 and the Sando host (GoingsOn ea517036). Design: wiki sando-mt-pom-pipelines.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-03 02:34 UTC
Signed with PGP, not checked
Commit: be3070c02b5ce70e3a71c82e17a2690829305fca
Parent: 95308d5
16 files changed, +654 insertions, -166 deletions
M multithreaded/CONTRIBUTING.md +16 -4
@@ -203,15 +203,27 @@
203 203
204 204 ## Deployment
205 205
206 - MT builds natively on Astra (aarch64). Deploy from `multithreaded/`:
206 + Full procedure: `deploy/README.md`.
207 +
208 + Production is not deployed from here. MT ships as a Sando companion of the MNW
209 + server: same worktree, same sha, same release bundle, installed on testnot-1 and
210 + prod-1 after the server's swap. A normal MNW deploy carries it, and it inherits
211 + the whole gate ladder (native build, migration dry run against mt's own dump,
212 + tests, burn-in, node health). `deploy/deploy-hetzner.sh`, which cross-compiled on
213 + macOS and scp'd to `root@`, is gone.
214 +
215 + The astra instance (aarch64) still has its own script, because Sando builds only
216 + on x86_64 fw13:
207 217
208 218 ```bash
209 219 ./deploy/deploy.sh # rsync source + shared deps, build on Astra, deploy binary + assets
210 220 ```
211 221
212 - The deploy script:
213 222 1. Rsyncs source to `~/src/multithreaded/` on Astra
214 - 2. Rsyncs shared deps (docengine, tagtree, s3-storage) to `~/src/shared/`
223 + 2. Rsyncs shared deps (tagtree, s3-storage) to `~/src/shared/`
215 224 3. Builds release binary on Astra
216 - 4. Copies binary + static + migrations to `/opt/multithreaded/`
225 + 4. Copies binary + migrations to `/opt/multithreaded/`
217 226 5. Restarts the systemd service
227 +
228 + Static assets are not copied by either path — they are compiled into the binary
229 + (`src/static_assets.rs`), so editing CSS or TypeScript needs a rebuild.
M multithreaded/Cargo.lock +20
@@ -1910,6 +1910,25 @@
1910 1910 "icu_properties",
1911 1911 ]
1912 1912
1913 + [[package]]
1914 + name = "include_dir"
1915 + version = "0.7.4"
1916 + source = "registry+https://github.com/rust-lang/crates.io-index"
1917 + checksum = "923d117408f1e49d914f1a379a309cffe4f18c05cf4e3d12e613a15fc81bd0dd"
1918 + dependencies = [
1919 + "include_dir_macros",
1920 + ]
1921 +
1922 + [[package]]
1923 + name = "include_dir_macros"
1924 + version = "0.7.4"
1925 + source = "registry+https://github.com/rust-lang/crates.io-index"
1926 + checksum = "7cab85a7ed0bd5f0e76d93846e0147172bed2e2d3f859bcc33a8d9699cad1a75"
1927 + dependencies = [
1928 + "proc-macro2",
1929 + "quote",
1930 + ]
1931 +
1913 1932 [[package]]
1914 1933 name = "indexmap"
1915 1934 version = "2.13.0"
@@ -2236,6 +2255,7 @@
2236 2255 "hex",
2237 2256 "hmac 0.13.0",
2238 2257 "http-body-util",
2258 + "include_dir",
2239 2259 "mt-core",
2240 2260 "mt-db",
2241 2261 "pom-contract",
M multithreaded/Cargo.toml +1
@@ -100,6 +100,7 @@
100 100 regex-lite = "0.1"
101 101 urlencoding = "2"
102 102 time = "0.3"
103 + include_dir = "0.7.4"
103 104
104 105 [dev-dependencies]
105 106 http-body-util = "0.1"
M multithreaded/build.rs +14
@@ -47,6 +47,20 @@
47 47 /// The generated partials are gitignored build output. They are written only
48 48 /// when their content changes, so a no-op build does not retrigger Askama.
49 49 fn fingerprint_assets() {
50 + // These watches now do double duty. They have always decided when the `?v=`
51 + // hash is recomputed; since `static/` is compiled into the binary
52 + // (src/static_assets.rs), they also decide when the embed is refreshed. An
53 + // asset outside this set is an asset that can go stale in the binary.
54 + //
55 + // `static/fonts` is watched as a directory — cargo walks it — and is safe to
56 + // watch because nothing in the build writes there.
57 + //
58 + // `static/dist` is deliberately NOT watched: `build_frontend` rewrites it on
59 + // every run, so watching it would rerun this script (and npm) on every
60 + // build. It is covered transitively instead, by the `frontend/src` watch in
61 + // `build_frontend` — a build-script rerun recompiles the crate, which is
62 + // when `include_dir!` re-reads the emitted bundles.
63 + println!("cargo::rerun-if-changed=static/fonts");
50 64 let static_files = ["static/style.css", "static/htmx.min.js", "static/mt.js"];
51 65
52 66 let mut hasher = DefaultHasher::new();
M sando/sando.toml +20
@@ -145,6 +145,16 @@
145 145 # binary can satisfy between restarts. The node binds HOST=127.0.0.1 PORT=8080
146 146 # (/etc/mnw/makenotwork.env); verified serving 200 on 2026-07-21.
147 147 health_url = "http://127.0.0.1:8080/health"
148 + # Companion: multithreaded (the forum) is built from this same sha and installed
149 + # after the server is up. It is on testnot FIRST, unlike mnw-cli, because this is
150 + # the tier where the 48h burn-in runs — a forum that fails to boot on the new
151 + # binary should fail here, not on prod. Needs the node-side wrapper + sudoers
152 + # grant (deploy/install-companion.sh, deploy/mnw-companion.sudoers) and an
153 + # existing multithreaded.service; see multithreaded/deploy/README.md.
154 + [[tier.node.companion]]
155 + name = "multithreaded"
156 + install_path = "/opt/multithreaded/multithreaded"
157 + service_name = "multithreaded.service"
148 158
149 159 # ---- B: prod-1 ----
150 160 [[tier]]
@@ -183,6 +193,16 @@
183 193 name = "mnw-cli"
184 194 install_path = "/opt/mnw-cli/mnw-cli"
185 195 service_name = "mnw-cli.service"
196 + # Companion: multithreaded, same sha as the server, same host. This is what
197 + # retires multithreaded/deploy/deploy-hetzner.sh (cross-compiled on macOS, scp'd
198 + # to root@, 41 commits behind when found). mt now rides the whole ladder — native
199 + # build on fw13, the migration_dry_run gate against its own dump, burn-in on
200 + # testnot, node_health — instead of a laptop and an scp. Design:
201 + # wiki sando-mt-pom-pipelines.
202 + [[tier.node.companion]]
203 + name = "multithreaded"
204 + install_path = "/opt/multithreaded/multithreaded"
205 + service_name = "multithreaded.service"
186 206
187 207 # ---- C: prod-2 (declared, not yet provisioned) ----
188 208 [[tier]]
M multithreaded/deploy/deploy.sh +4 -2
@@ -56,13 +56,15 @@
56 56 echo "[build] Done"
57 57 }
58 58
59 + # static/ is deliberately not copied: it is compiled into the binary
60 + # (src/static_assets.rs), so the binary IS the assets. migrations/ is copied for
61 + # operator convenience only — sqlx::migrate!() embeds them at compile time too.
59 62 deploy_files() {
60 - echo "[deploy] Stopping service + copying binary + assets to $REMOTE_DIR..."
63 + echo "[deploy] Stopping service + copying binary to $REMOTE_DIR..."
61 64 ssh $SERVER "
62 65 sudo systemctl stop multithreaded || true
63 66 sudo cp ~/$SRC_DIR/target/release/$BINARY_NAME $REMOTE_DIR/$BINARY_NAME
64 67 sudo chmod +x $REMOTE_DIR/$BINARY_NAME
65 - sudo rsync -a --delete ~/$SRC_DIR/static/ $REMOTE_DIR/static/
66 68 sudo rsync -a --delete ~/$SRC_DIR/migrations/ $REMOTE_DIR/migrations/
67 69 sudo chown -R multithreaded:multithreaded $REMOTE_DIR
68 70 "
M multithreaded/docs/oauth_flow.md +1 -1
@@ -205,4 +205,4 @@
205 205 - `src/csrf.rs`: CSRF token generation, middleware, constant-time comparison
206 206 - `src/internal_auth.rs`: HMAC-SHA256 auth for MNW-to-MT internal API (separate from OAuth)
207 207 - `src/main.rs`: session store setup, session layer config, middleware stack
208 - - `deploy/deploy-hetzner.sh`, `deploy/deploy.sh`: deploy scripts (production env vars are provisioned on-server, not committed to the repo)
208 + - `deploy/README.md`, `deploy/deploy.sh`: deploy procedure (production env vars are provisioned on-server, not committed to the repo)
M multithreaded/src/lib.rs +1
@@ -15,6 +15,7 @@
15 15 pub mod maintenance;
16 16 pub mod routes;
17 17 pub mod seed;
18 + pub mod static_assets;
18 19 pub mod storage;
19 20 pub mod templates;
20 21 pub mod trusted_proxy;
M multithreaded/src/main.rs +8 -2
@@ -1,7 +1,6 @@
1 1 use multithreaded::{AppState, config::Config, csrf};
2 2 use sqlx::postgres::PgPoolOptions;
3 3 use tokio::net::TcpListener;
4 - use tower_http::services::ServeDir;
5 4 use tower_sessions::ExpiredDeletion;
6 5 use tower_sessions::SessionManagerLayer;
7 6 use tower_sessions::cookie::SameSite;
@@ -140,7 +139,14 @@
140 139 let app = forum
141 140 // Internal API routes, HMAC auth only, no CSRF/session middleware
142 141 .merge(multithreaded::routes::internal::internal_routes(state))
143 - .nest_service("/static", ServeDir::new("static"))
142 + // Assets are compiled into the binary, not read from a `static/` dir
143 + // beside it, so mt ships as the single file Sando's companion mechanism
144 + // installs and its markup can never disagree with its stylesheet. See
145 + // multithreaded::static_assets.
146 + .route(
147 + "/static/{*path}",
148 + axum::routing::get(multithreaded::static_assets::serve),
149 + )
144 150 .layer(tower_http::set_header::SetResponseHeaderLayer::overriding(
145 151 axum::http::header::CONTENT_SECURITY_POLICY,
146 152 axum::http::HeaderValue::from_static(
M sando/daemon/sando-daemon.toml +19
@@ -70,6 +70,25 @@
70 70 manifest_dir = "mnw-cli"
71 71 bin = "mnw-cli"
72 72
73 + # multithreaded (forums.makenot.work) runs on alpha-west-1, the same host as the
74 + # server, x86_64, so the build host can compile it and the never-cross-compile
75 + # rule is satisfied. Its own deploy path was multithreaded/deploy/deploy-hetzner.sh,
76 + # which cross-compiled on macOS via cargo zigbuild and scp'd to root@ — against
77 + # that rule, and 41 commits behind by the time it was found (2026-07-30).
78 + #
79 + # mt is a single file to install because it embeds its own `static/` tree
80 + # (multithreaded/src/static_assets.rs). It used to serve those assets off disk
81 + # beside the binary, which the one-file companion mechanism cannot ship.
82 + #
83 + # Note this is a companion and NOT an entry in `bin_names`: bin_names lists the
84 + # bin targets of the SERVER crate, resolved under its own target/release, so
85 + # naming multithreaded there would fail the build looking for a binary the
86 + # server never produces. Wiki sando-mt-pom-pipelines had it wrong.
87 + [[companion]]
88 + name = "multithreaded"
89 + manifest_dir = "multithreaded"
90 + bin = "multithreaded"
91 +
73 92 # Crates the cargo_test gate runs, in order. This list used to be hardcoded to
74 93 # `server`, so every other crate in the repo shipped ungated — including
75 94 # mnw-cli, which is BUILT as a companion and installed onto prod-1 in the same
M sando/deploy/sando-daemon.toml.example +98
@@ -71,6 +71,104 @@
71 71 [[frontend_build]]
72 72 dir = "multithreaded/frontend"
73 73
74 + # Crates built from the same worktree/sha as the server and staged into the
75 + # release bundle as `companions/<name>`, so a service that shares the server's
76 + # contract cannot drift out of lockstep. This list only says what to BUILD and
77 + # stage; which nodes install which companion is per-node in the topology (see
78 + # the [[tier.node.companion]] blocks in sando.toml).
79 + #
80 + # mnw-cli is the public git-SSH server that proxies to /api/internal/* — a
81 + # two-month drift there is what broke git hosting during the 0.10.14 deploy.
82 + [[companion]]
83 + name = "mnw-cli"
84 + manifest_dir = "mnw-cli"
85 + bin = "mnw-cli"
86 +
87 + # multithreaded (forums.makenot.work) runs on alpha-west-1, the same host as the
88 + # server and the same architecture, so the build host can compile it. Replaces
89 + # multithreaded/deploy/deploy-hetzner.sh, which cross-compiled on macOS via
90 + # cargo zigbuild and scp'd to root@ — against the native-build rule, and the
91 + # real deploy path until 2026-08-02.
92 + #
93 + # One file to install: mt embeds its own static/ tree
94 + # (multithreaded/src/static_assets.rs) rather than serving it from a directory
95 + # beside the binary, which is what makes it fit a mechanism that ships exactly
96 + # one file per companion.
97 + #
98 + # NOT a `bin_names` entry — that list is the SERVER crate's bin targets.
99 + [[companion]]
100 + name = "multithreaded"
101 + manifest_dir = "multithreaded"
102 + bin = "multithreaded"
103 +
104 + # Crates the cargo_test gate runs, in order; the gate stops at the first red one.
105 + # This list used to be absent here, which means the default applied — a single
106 + # `server` entry — so everything else in the repo shipped ungated, the companions
107 + # installed onto prod-1 included. A target with no Cargo.toml in the sha being
108 + # built is skipped with a warning rather than failing, so this list can describe
109 + # the tip and still bisect.
110 + #
111 + # scratch_db exports DATABASE_URL/TEST_DATABASE_URL. Only crates whose sqlx
112 + # macros type-check against a live DB need it; the rest ship offline `.sqlx` data
113 + # and are left in offline mode.
114 + [[test_target]]
115 + dir = "server"
116 + features = ["fast-tests"]
117 + scratch_db = true
118 +
119 + # Companion: ships to prod-1, so it is gated first among the rest.
120 + [[test_target]]
121 + dir = "mnw-cli"
122 +
123 + # Companion: ships to testnot-1 and prod-1 (see sando.toml).
124 + [[test_target]]
125 + dir = "multithreaded"
126 + scratch_db = true
127 +
128 + [[test_target]]
129 + dir = "pom"
130 +
131 + [[test_target]]
132 + dir = "wam"
133 +
134 + # shared/* — every app and service in the repo links these.
135 + #
136 + # No `shared/docengine` entry: docengine left the repo for Libraries/docengine on
137 + # 2026-07-30 and is consumed as a path dep resolved through the [[aux_repo]]
138 + # checkout, which is beside the worktree rather than under it. test_target dirs
139 + # are worktree-relative, so it is not addressable from here. (The dev config at
140 + # daemon/sando-daemon.toml still lists it; that entry is a warn-and-skip no-op.)
141 + [[test_target]]
142 + dir = "shared/egui-updater"
143 +
144 + [[test_target]]
145 + dir = "shared/kberg"
146 + all_features = true
147 +
148 + [[test_target]]
149 + dir = "shared/livechat"
150 + all_features = true
151 +
152 + [[test_target]]
153 + dir = "shared/ops-core"
154 +
155 + [[test_target]]
156 + dir = "shared/ops-exec"
157 + all_features = true
158 +
159 + [[test_target]]
160 + dir = "shared/pom-contract"
161 +
162 + [[test_target]]
163 + dir = "shared/s3-storage"
164 +
165 + [[test_target]]
166 + dir = "shared/synckit-client"
167 + all_features = true
168 +
169 + [[test_target]]
170 + dir = "shared/tagtree"
171 +
74 172 # Non-binary content shipped as part of each release. Multiple entries can
75 173 # target the same `dst` (additive merge — used to build `docs/` from three
76 174 # worktree sources). Sources are relative to the worktree root; dsts are
M sando/daemon/src/main.rs +18 -12
@@ -50,6 +50,7 @@
50 50 let topo = topology::Topology::load(&cfg.topology_path)?;
51 51 topo.ensure_build_host_not_serving(&cfg.build_host)?;
52 52 topo.ensure_migration_checks_have_backups(&cfg.migration_checks)?;
53 + topo.ensure_node_companions_are_built(&cfg.companions)?;
53 54 Ok(topo)
54 55 }
55 56
@@ -73,6 +74,7 @@
73 74 let topo = Arc::new(topology::Topology::load(&cfg.topology_path)?);
74 75 topo.ensure_build_host_not_serving(&cfg.build_host)?;
75 76 topo.ensure_migration_checks_have_backups(&cfg.migration_checks)?;
77 + topo.ensure_node_companions_are_built(&cfg.companions)?;
76 78 tokio::fs::create_dir_all(&cfg.workdir).await?;
77 79 tokio::fs::create_dir_all(&cfg.release_root).await?;
78 80 git::ensure_bare_repo(Path::new(&topo.repo.bare_path)).await?;
@@ -180,18 +182,22 @@
180 182 // binary must pass against the topology this build ships with — the
181 183 // regression guard for the 2026-07-09 brick, where a binary that could
182 184 // not parse the live node_health config was installed and crash-looped.
183 - // (`Config::for_tests` is lib-test-only and unavailable here, so build
184 - // the config by deserializing, pointing it at the shipped sando.toml.)
185 - let topo_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../sando.toml");
186 - let raw = format!(
187 - "listen = \"127.0.0.1:7766\"\n\
188 - db_path = \"./sando.db\"\n\
189 - topology_path = {topo_path:?}\n\
190 - build_host = \"fw13\"\n\
191 - workdir = \"./work\"\n\
192 - release_root = \"./releases\"\n"
193 - );
194 - let cfg: super::config::Config = toml::from_str(&raw).expect("build fixture daemon config");
185 + // Both halves of the pair are the shipped ones. This used to hand-roll a
186 + // minimal daemon config, which meant the production daemon config was
187 + // never validated by anything and the cross-file checks had nothing real
188 + // to check: the fixture declared no companions while the shipped
189 + // topology installs two, so `ensure_node_companions_are_built` would
190 + // have passed on a fixture and failed on the box.
191 + //
192 + // Only `topology_path` is overridden, because the shipped value is the
193 + // installed location (/etc/sando/sando.toml) and the repo copy is what
194 + // this build ships.
195 + let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
196 + let raw = std::fs::read_to_string(root.join("../deploy/sando-daemon.toml.example"))
197 + .expect("reading the shipped production daemon config");
198 + let mut cfg: super::config::Config =
199 + toml::from_str(&raw).expect("shipped daemon config must parse");
200 + cfg.topology_path = root.join("../sando.toml");
195 201 validate_loaded(&cfg).expect("shipped config must pass --check-config");
196 202 }
197 203
M sando/daemon/src/topology.rs +122
A multithreaded/deploy/README.md +67
D multithreaded/deploy/deploy-hetzner.sh -145
A multithreaded/src/static_assets.rs +245