Skip to main content

max / makenotwork

20.0 KB · 410 lines History Blame Raw
1 # sando
2
3 Home-rolled CI/CD controller for the MNW server. Axum daemon (`sandod`) +
4 ratatui TUI (`sando`). Gates a tiered deploy flow:
5
6 ```
7 git push mm -> MakeMachine (build + tests + migration dry-run + boot smoke)
8 -> A (testnot.work)
9 -> B (prod-1)
10 -> C (prod-2)
11 ```
12
13 Each tier's progression gates are declared in `sando.toml`. Tiers and nodes
14 live in the TOML, not in code — adding a node or a new tier is a config edit.
15
16 ## Crates
17
18 | Path | Binary | Role |
19 |------|--------|------|
20 | `daemon/` | `sandod` | Axum daemon. Runs on the MakeMachine. Owns SQLite state, the bare git repo, and all build/gate/deploy logic. |
21 | `tui/` | `sando` | ratatui front-end. Runs on the laptop. Talks to `sandod` over the tailnet. |
22
23 ## Quickstart: localhost dev loop
24
25 The MakeMachine hardware does not exist yet, so v0 runs entirely on a single
26 host. Bare repo, releases dir, "remote" A node — everything is a local
27 directory.
28
29 ```bash
30 # 1. Build both binaries.
31 cd MNW/sando/daemon && cargo build
32 cd ../tui && cargo build
33
34 # 2. Create a workspace + config.
35 mkdir -p /tmp/sando-dev
36 cat > /tmp/sando-dev/daemon.toml <<EOF
37 listen = "127.0.0.1:7766"
38 db_path = "/tmp/sando-dev/sando.db"
39 topology_path = "/tmp/sando-dev/sando.toml"
40 workdir = "/tmp/sando-dev/work"
41 release_root = "/tmp/sando-dev/releases"
42 # scratch_db_url = "postgres://you@127.0.0.1/sando_scratch"
43 EOF
44
45 cat > /tmp/sando-dev/sando.toml <<EOF
46 [repo]
47 bare_path = "/tmp/sando-dev/mnw.git"
48 branch = "main"
49 [backup]
50 source = "file:///tmp/sando-dev/fake-backup.sql"
51 local_path = "/tmp/sando-dev/backup.sql"
52
53 [[tier]]
54 name = "mm"
55 provisioned = true
56 gates = [
57 { kind = "cargo_test" },
58 { kind = "migration_dry_run" },
59 { kind = "boot_smoke" },
60 ]
61
62 [[tier]]
63 name = "a"
64 provisioned = true
65 canary = "sequential"
66 gates = [
67 { kind = "boot_smoke" },
68 { kind = "manual_confirm" },
69 ]
70 [[tier.node]]
71 name = "a-local"
72 ssh_target = "local"
73 release_root = "/tmp/sando-dev/a-node"
74 EOF
75
76 # 3. Run the daemon.
77 SANDO_CONFIG=/tmp/sando-dev/daemon.toml \
78 ./MNW/sando/daemon/target/debug/sandod
79
80 # 4. In another shell: point a clone at the bare repo and push.
81 git clone /tmp/sando-dev/mnw.git /tmp/sando-dev/checkout
82 # ... add a `server/Cargo.toml` + source so the build can run ...
83 cd /tmp/sando-dev/checkout && git push origin main
84
85 # 5. Watch the TUI.
86 SANDO_DAEMON=http://127.0.0.1:7766 ./MNW/sando/tui/target/debug/sando
87 ```
88
89 When you push, the bare repo's `post-receive` hook (installed automatically
90 by `sandod` on startup) calls `POST /rebuild`. The daemon checks out the
91 sha, runs `cargo build --release` against `server/`, stages the release bundle
92 under `staging/<build_id>/`, then runs the host tier's gates. On green the
93 bundle is published content-addressed at `releases/<digest16>/` (the sha256 of
94 its `MANIFEST` rather than the version; see `daemon/src/bundle.rs`), `current`
95 is swapped to it, and the host tier's `tier_state` advances. Promote with:
96
97 ```bash
98 curl -X POST http://127.0.0.1:7766/promote/a \
99 -H 'Content-Type: application/json' \
100 -d '{"version":"0.8.2"}'
101 ```
102
103 ## Gates
104
105 Build-time gates run on the host tier, once per build, against the worktree.
106 Promote-time gates run against a tier's deployed nodes or its operator.
107
108 | Kind | When | What it proves |
109 |------|------|----------------|
110 | `code_smoke` | build | Compiles every `[[frontend_build]]` (see below), then boots the fresh binary on a throwaway DB it migrates from scratch and seeds, then probes `/health`. Runs first: green here isolates a later red as an environment problem, not a code one. |
111 | `fmt` | build | `cargo fmt --check` over every `[[test_target]]`. No compilation, so it fails fast. |
112 | `cargo_test` | build | Every configured `[[test_target]]` crate's suite, in order (see below). |
113 | `hardening_test` | build | What `cargo_test` structurally cannot reach (see below). |
114 | `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. |
115 | `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. |
116 | `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. |
117 | `migration_dry_run` | build | Every configured `[[migration_check]]` database's migrations apply cleanly to a restored production dump of *that* database. Blocks if a check's newest fetched dump is older than `backup_max_age_hours` (default 48) — a stale dump proves nothing about today's schema. |
118 | `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. |
119 | `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. |
120 | `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. |
121 | `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. |
122
123 A tier's gates guard promotion **out** of it. So the gate list that stands
124 between staging and production is tier `a`'s, not tier `b`'s — sign-off for a
125 prod ship is `POST /confirm/a`, run after the version lands on `a` (the
126 confirmation must be fresher than that landing) and before `POST /promote/b`.
127
128 ### What `cargo_test` runs
129
130 The gate used to be hardcoded to `worktree/server`, so every other crate in the
131 repo shipped ungated. `mnw-cli` was the sharp edge: it is built as a companion
132 and installed onto prod-1 in the same promote that ships the server, with no
133 test ever run against it.
134
135 Targets are now configured in the daemon config:
136
137 ```toml
138 [[test_target]]
139 dir = "server"
140 features = ["fast-tests"]
141 scratch_db = true # export DATABASE_URL / TEST_DATABASE_URL
142
143 [[test_target]]
144 dir = "shared/tagtree" # no features, no DB
145
146 [[test_target]]
147 dir = "shared/ops-exec"
148 all_features = true # mutually exclusive with `features`
149 ```
150
151 Omitting the key entirely keeps the historical behavior: one `server` target
152 with `fast-tests`, against the scratch DB.
153
154 Notes on the semantics:
155
156 - `scratch_db` is opt-in per target. Setting `DATABASE_URL` takes sqlx **out**
157 of offline mode, so a crate that ships `.sqlx` query data would try to
158 type-check against a database holding none of its tables.
159 - All targets share one `gate_runs` row and one log file, sectioned by
160 `==== test_target: <dir> ====` banners. The gate stops at the first red
161 target, and the failure summary names the crate.
162 - `gate_timeout_secs` bounds the whole gate, not each target.
163 - A target whose directory is absent from the worktree is skipped with a
164 warning, so sando can still build older shas from a config describing the
165 tip. If *no* target exists, the gate fails rather than reporting a pass over
166 zero suites.
167
168 ### What `migration_dry_run` restores
169
170 The gate used to be hardcoded to `worktree/server/migrations` against the one
171 `scratch_db_url`, so it gated exactly one database. multithreaded ships its own
172 migrations and applies them with `sqlx::migrate!()` at boot against its own
173 database, which meant it carried the server's exposure with none of the server's
174 gate: an edited already-applied migration would not fail a dry run, it would fail
175 to boot in prod. (That is not hypothetical — an exorcise sweep rewrote comments
176 in 29 applied server migrations in July 2026, and sqlx checksums whole files. The
177 gate caught it. mt was spared by luck.)
178
179 Each database now gets a check in the daemon config, paired with its own dump in
180 the topology:
181
182 ```toml
183 # sando.toml (topology)
184 [[backup]]
185 name = "server"
186 source = "ssh://backup-puller@alpha-west-1:2200/makenotwork/latest.sql.gz"
187 local_path = "/srv/sando/backups/latest.sql.gz"
188
189 [[backup]]
190 name = "multithreaded"
191 source = "ssh://backup-puller@alpha-west-1:2200/multithreaded/latest.sql.gz"
192 local_path = "/srv/sando/backups/multithreaded-latest.sql.gz"
193 ```
194
195 ```toml
196 # sando-daemon.toml
197 [[migration_check]]
198 dir = "server/migrations"
199 backup = "server"
200
201 [[migration_check]]
202 dir = "multithreaded/migrations"
203 backup = "multithreaded"
204 scratch_db = "sando_scratch_mt"
205 owner_role = "multithreaded"
206 ```
207
208 Omitting either key keeps the historical behavior: one `server/migrations` check
209 against one `[backup]`, which still parses as a single-entry list.
210
211 Notes on the semantics:
212
213 - **A check restores its own database's dump.** Restoring the server's dump under
214 another service's migrations would fail on the first migration for the least
215 interesting reason — a `_sqlx_migrations` table full of someone else's rows.
216 - `scratch_db` is what keeps checks from clobbering each other. The server check
217 leaves it unset, so it runs against `scratch_db_url` itself and leaves it in
218 migrated state for `cargo_test` to reuse; every other check names its own
219 database, which the daemon drops and recreates at the start of the check. Two
220 checks sharing one is refused at config load.
221 - `owner_role` defaults to `scratch_owner_role`. A dump carries
222 `ALTER ... OWNER TO <role>` for every object, and the role has to exist in the
223 scratch cluster before the restore, so a dump owned by anyone else needs this.
224 - Freshness and the fetch's plausibility floor are both per-dump. A fresh server
225 dump does not make a 45-day-old mt dump look current, and the server's size
226 does not set mt's floor (they differ by two orders of magnitude).
227 - All checks share one `gate_runs` row and one log file, sectioned by
228 `==== migration_check: <dir> ====` banners. The gate stops at the first red
229 check.
230 - A `[[migration_check]]` naming a `backup` the topology does not declare fails
231 at startup, not at the first promote.
232
233 ### What `code_smoke` builds first
234
235 Before it creates a database or boots anything, `code_smoke` compiles every
236 configured frontend:
237
238 ```toml
239 [[frontend_build]]
240 dir = "server/frontend"
241
242 [[frontend_build]]
243 dir = "multithreaded/frontend" # script = "build" by default
244 ```
245
246 These are npm projects whose compiled output the binary serves but whose failure
247 `cargo build` will not report. Both MNW crates compile TypeScript from a build
248 script that downgrades a `tsc` error to a `cargo::warning` and lets the Rust
249 build succeed against whatever `static/dist/` already holds, on purpose, so a
250 type error in a chat widget cannot stop the forum from compiling. The cost is
251 that nothing downstream noticed either, and the deploy rsynced the previous
252 build's bundle. This is the one place that failure is fatal.
253
254 Semantics match `cargo_test`: `npm ci` first if `node_modules` is absent (usually
255 it is not, because the build script that produced the artifact already installed
256 it), stop at the first red project with the directory named, one deadline across
257 the gate, and a project absent from the worktree is skipped with a log line so
258 older shas still rebuild. Omitting the key entirely gates nothing, which is the
259 right default for a project with no frontend.
260
261 ### The lint and supply-chain gates
262
263 `clippy` and `fmt` run over the same `[[test_target]]` list as `cargo_test`,
264 with the same semantics: per-target log banners, stop at the first red target
265 with the crate named, one deadline across the whole gate.
266
267 `cargo_audit` and `cargo_deny` are **config-gated**: a target only qualifies
268 once it carries a `.cargo/audit.toml` or `deny.toml`. Both tools are only
269 meaningful against a triaged posture, and four crates in this repo fail
270 `cargo audit` purely for lack of a file recording which transitive advisories
271 have been reviewed and accepted. Running them everywhere would make the gate
272 permanently and uninformatively red, which teaches everyone to ignore it.
273 Dropping the config file into a crate is what opts it in.
274
275 Before these existed, `-D warnings` was enforced in exactly one place
276 (`server/deploy/run-ci.sh`, which died with the astra pipeline) and
277 `cargo fmt --check` nowhere at all.
278
279 ### Why `hardening_test` exists
280
281 `cargo_test` builds with `--features fast-tests`, which relaxes
282 `AUTH_RATE_LIMIT_BURST` (5 to 20), `SANDBOX_RATE_LIMIT_MS` (30s to 10ms), and
283 argon2 (46 MiB/t=2 down to 8 MiB/t=1) so the signup-heavy workflow suite
284 finishes in reasonable time. On top of that, the rate-limiting tests are
285 `#[cfg_attr(feature = "fast-tests", ignore)]`d, because a bucket refilling at
286 100/sec never depletes under parallel test threads.
287
288 The net effect was that Sando's only code gate skipped every test of the auth
289 hardening it exists to protect. `hardening_test` re-runs that suite with no
290 features, single-threaded, against production constants. It costs a second
291 compile of the server's test binary, since a different feature set is a
292 different cfg and shares no artifacts. It also fails closed if its name filter
293 matches zero tests, so renaming the suite cannot quietly turn the gate into a
294 green no-op.
295
296 ## API
297
298 | Method | Path | Body | Purpose |
299 |--------|------|------|---------|
300 | GET | `/state` || Tier list + current/previous version + last gate outcomes, plus `build` (latest build run: phase/result/failure_summary/elapsed_s, `null` until first `/rebuild`) so a poller sees in-flight/failed builds, not a frozen version |
301 | POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). Returns `{accepted, sha, run_id}`. |
302 | POST | `/intake` | `{staged, record}` | Accept an artifact built elsewhere and take it through the same host-tier gating a locally-built one gets. `staged` is a directory already under this app's `release_root/staging/`; `record` is the builder's `ArtifactRecord` verbatim. The bytes are proved against the record before anything else happens — a bundle that drifted in transit is refused with the offending file named. Returns `{accepted, run_id}`. |
303 | GET | `/runs/{id}` || Build-status of the run a `/rebuild` returned: `{run_id, sha, version, phase, result, failure_summary, gates[], started_at, finished_at}`. The pollable resource for a non-TUI driver — `/state` only reflects the last *successful* version. |
304 | GET | `/runs/{id}/wait` | `?timeout_ms=` | Long-poll: blocks until the run settles or `timeout_ms` (default 30s, cap 120s) elapses, then returns the same `RunView`. Fire `/rebuild` → block on `/wait`. |
305 | POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial` — the rollout landed, the tier cannot promote onward. |
306 | POST | `/rollback/{tier}` || Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. |
307 | POST | `/confirm/{tier}` || Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. |
308 | POST | `/backup/fetch` | `{force?, name?}` | Pull every configured prod dump, or just `name`. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one *of the same name*, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. Each dump is attempted even if another fails. |
309 | GET | `/events` || WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). |
310
311 ## TUI
312
313 `sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings:
314
315 | key | action |
316 |-----|--------|
317 | ↑/↓ or j/k | select tier |
318 | p | `POST /promote/<selected>` (no body — version defaults to predecessor's current) |
319 | R | `POST /rollback/<selected>` |
320 | b | `POST /backup/fetch` |
321 | c | `POST /confirm/<selected>` |
322 | r | refresh hint (poller is already every 2s) |
323 | q / Esc / Ctrl-C | quit |
324
325 Action results show up in the events log a moment later (the actions themselves emit events from the daemon side).
326
327 ## Hotfix flow
328
329 `POST /promote/{tier}` accepts:
330
331 - `hotfix: true` — skips the `burn_in` gate on the predecessor tier only. All
332 other gates still apply, `manual_confirm` included: a hotfix still needs an
333 operator sign-off before it reaches production.
334 - `reset_burn_in: true` (default `false`) — additionally nulls
335 `tier_state.burn_in_started_at` on the source tier, restarting the clock
336 for whatever else is still burning in there. Use this only when the hotfix
337 meaningfully changes the surface area under burn-in.
338
339 ## Shipping more than one product
340
341 One daemon, one database, one bind address — and beneath that, N independent
342 pipelines, each with its own repo, tiers, nodes, gates, release root and version
343 history. `[app.<id>]` in `sando-daemon.toml` points at each product's own config;
344 a file with no `[app.*]` tables is read as the single app `mnw`, so a config
345 written before any of this still loads.
346
347 The unprefixed routes address the default product, because `/promote/b` is what
348 the runbook says and what an operator types under pressure. Every product is also
349 at `/apps/<id>/...`, and `GET /apps` reports what is mounted.
350
351 **A product does not have to be one Sando builds.** `pom` is the first that is
352 not: it runs on aarch64 and on x86_64, and Sando compiles on one configured host,
353 so it could never build half of a pom release without breaking its own
354 never-cross-compile rule. Bento builds it natively on both; Sando gates what
355 arrives and performs the advance. Such a product declares no `build_host` and no
356 `[repo]` — absent is not "build anywhere", it is a statement that Sando does not
357 build this at all, and `/rebuild` refuses rather than choosing a machine.
358
359 ### Platforms, and why a bundle cannot land on the wrong box
360
361 One pom version is two bundles with two digests. Which one a node gets is not a
362 check before the deploy call — it is the only way to make the call:
363
364 ```rust
365 let placement = Placement::check(node, bundle, artifact_platform)?;
366 deploy_node(executor, placement, version, primary_bin).await
367 ```
368
369 `deploy_node` takes a `Placement`, and `Placement::check` is its only
370 constructor, so a mismatched deploy is not a bug to avoid but a value that
371 cannot be built. Both sides state a platform (`platform = "linux/aarch64"` on a
372 node, the artifact record's provenance for a bundle) and they must be equal.
373 Silence on one side is a refusal, not a pass; both silent is the single-platform
374 world MNW still lives in, and the moment either side starts stating, the other
375 has to as well.
376
377 Promote resolves per node before any node is touched, so a version missing its
378 x86_64 half fails whole rather than halfway down a rollout. A sibling bundle only
379 qualifies if its own run settled green: each architecture stands on its own
380 intake and its own gate run, because the source tier's evidence says nothing
381 about bytes it never saw.
382
383 ### Which gates go where
384
385 Evidence *about the artifact*`cargo_test`, `clippy`, `fmt`, the audits —
386 belongs to the builder. Evidence *about the artifact in an environment*
387 `migration_dry_run`, `boot_smoke`, `node_health`, `burn_in`, `manual_confirm`
388 is Sando's, which also keeps production dumps on the machine that already has
389 them instead of handing them to build hosts.
390
391 An accepted artifact has no worktree, so a source-reading gate configured on its
392 tier refuses rather than resolving against nothing and reporting green.
393 `migration_dry_run` resolves its migrations from the bundle first and the
394 worktree second, which is why MNW now stages `server/migrations` and
395 `multithreaded/migrations` into the bundle: inside the digest, the gate proves
396 something about the bytes that ship rather than about a checkout sitting beside
397 them.
398
399 ## v0 limitations
400
401 - `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The
402 gate drops every non-system schema on every run; do not point this at
403 anything that matters. A check with its own `scratch_db` gets that whole
404 database dropped and recreated instead.
405
406 ## License
407
408 MIT. The surrounding MNW monorepo is PolyForm-Noncommercial — sando is
409 deliberately MIT'd because it's deploy infra, not the product.
410