Skip to main content

max / makenotwork

19.1 KB · 396 lines History Blame Raw
1 # sando
2
3 Home-rolled CI/CD controller for the MNW server. Axum daemon (`sandod`) +
4 ratatui TUI (`sando`). Gates a tiered deploy flow:
5
6 ```
7 git push mm -> MakeMachine (build + tests + migration dry-run + boot smoke)
8 -> A (testnot.work)
9 -> B (prod-1)
10 -> C (prod-2)
11 ```
12
13 Each tier's progression gates are declared in `sando.toml`. Tiers and nodes
14 live in the TOML, not in code; adding a node or a new tier is a config edit.
15
16 ## Crates
17
18 | Path | Binary | Role |
19 |------|--------|------|
20 | `daemon/` | `sandod` | Axum daemon. Runs on the MakeMachine. Owns SQLite state, the bare git repo, and all build/gate/deploy logic. |
21 | `tui/` | `sando` | ratatui front-end. Runs on the laptop. Talks to `sandod` over the tailnet. |
22
23 ## Quickstart: localhost dev loop
24
25 The MakeMachine hardware does not exist yet, so v0 runs entirely on a single
26 host. Bare repo, releases dir, "remote" A node — everything is a local
27 directory.
28
29 ```bash
30 # 1. Build both binaries.
31 cd MNW/sando/daemon && cargo build
32 cd ../tui && cargo build
33
34 # 2. Create a workspace + config.
35 mkdir -p /tmp/sando-dev
36 cat > /tmp/sando-dev/daemon.toml <<EOF
37 listen = "127.0.0.1:7766"
38 db_path = "/tmp/sando-dev/sando.db"
39 topology_path = "/tmp/sando-dev/sando.toml"
40 workdir = "/tmp/sando-dev/work"
41 release_root = "/tmp/sando-dev/releases"
42 # scratch_db_url = "postgres://you@127.0.0.1/sando_scratch"
43 EOF
44
45 cat > /tmp/sando-dev/sando.toml <<EOF
46 [repo]
47 bare_path = "/tmp/sando-dev/mnw.git"
48 branch = "main"
49 [backup]
50 source = "file:///tmp/sando-dev/fake-backup.sql"
51 local_path = "/tmp/sando-dev/backup.sql"
52
53 [[tier]]
54 name = "mm"
55 provisioned = true
56 gates = [
57 { kind = "cargo_test" },
58 { kind = "migration_dry_run" },
59 { kind = "boot_smoke" },
60 ]
61
62 [[tier]]
63 name = "a"
64 provisioned = true
65 canary = "sequential"
66 gates = [
67 { kind = "boot_smoke" },
68 { kind = "manual_confirm" },
69 ]
70 [[tier.node]]
71 name = "a-local"
72 ssh_target = "local"
73 release_root = "/tmp/sando-dev/a-node"
74 EOF
75
76 # 3. Run the daemon.
77 SANDO_CONFIG=/tmp/sando-dev/daemon.toml \
78 ./MNW/sando/daemon/target/debug/sandod
79
80 # 4. In another shell: point a clone at the bare repo and push.
81 git clone /tmp/sando-dev/mnw.git /tmp/sando-dev/checkout
82 # ... add a `server/Cargo.toml` + source so the build can run ...
83 cd /tmp/sando-dev/checkout && git push origin main
84
85 # 5. Watch the TUI.
86 SANDO_DAEMON=http://127.0.0.1:7766 ./MNW/sando/tui/target/debug/sando
87 ```
88
89 When you push, the bare repo's `post-receive` hook (installed automatically
90 by `sandod` on startup) calls `POST /rebuild`. The daemon checks out the
91 sha, runs `cargo build --release` against `server/`, stages the release bundle
92 under `staging/<build_id>/`, then runs the host tier's gates. On green the
93 bundle is published content-addressed at `releases/<digest16>/` (the sha256 of
94 its `MANIFEST` rather than the version; see `daemon/src/bundle.rs`), `current`
95 is swapped to it, and the host tier's `tier_state` advances. Promote with:
96
97 ```bash
98 curl -X POST http://127.0.0.1:7766/promote/a \
99 -H 'Content-Type: application/json' \
100 -d '{"version":"0.8.2"}'
101 ```
102
103 ## Gates
104
105 Build-time gates run on the host tier, once per build, against the worktree.
106 Promote-time gates run against a tier's deployed nodes or its operator.
107
108 | Kind | When | What it proves |
109 |------|------|----------------|
110 | `code_smoke` | build | Compiles every `[[frontend_build]]` (see below), then boots the fresh binary on a throwaway DB it migrates from scratch and seeds, then probes `/health`. Runs first: green here isolates a later red as an environment problem, not a code one. |
111 | `fmt` | build | `cargo fmt --check` over every `[[test_target]]`. No compilation, so it fails fast. |
112 | `cargo_test` | build | Every configured `[[test_target]]` crate's suite, in order (see below). |
113 | `hardening_test` | build | What `cargo_test` structurally cannot reach (see below). |
114 | `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. |
115 | `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. |
116 | `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. |
117 | `migration_dry_run` | build | Every configured `[[migration_check]]` database's migrations apply cleanly to a restored production dump of *that* database. Blocks if a check's newest fetched dump is older than `backup_max_age_hours` (default 48); a stale dump proves nothing about today's schema. |
118 | `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. |
119 | `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. |
120 | `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. |
121 | `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. |
122
123 A tier's gates guard promotion **out** of it. So the gate list that stands
124 between staging and production is tier `a`'s, not tier `b`'s. Sign-off for a
125 prod ship is `POST /confirm/a`, run after the version lands on `a` (the
126 confirmation must be fresher than that landing) and before `POST /promote/b`.
127
128 ### What `cargo_test` runs
129
130 Targets are configured in the daemon config:
131
132 ```toml
133 [[test_target]]
134 dir = "server"
135 features = ["fast-tests"]
136 scratch_db = true # export DATABASE_URL / TEST_DATABASE_URL
137
138 [[test_target]]
139 dir = "shared/tagtree" # no features, no DB
140
141 [[test_target]]
142 dir = "shared/ops-exec"
143 all_features = true # mutually exclusive with `features`
144 ```
145
146 Omitting the key entirely defaults to one `server` target with `fast-tests`,
147 against the scratch DB.
148
149 Notes on the semantics:
150
151 - `scratch_db` is opt-in per target. Setting `DATABASE_URL` takes sqlx **out**
152 of offline mode, so a crate that ships `.sqlx` query data would try to
153 type-check against a database holding none of its tables.
154 - All targets share one `gate_runs` row and one log file, sectioned by
155 `==== test_target: <dir> ====` banners. The gate stops at the first red
156 target, and the failure summary names the crate.
157 - `gate_timeout_secs` bounds the whole gate, not each target.
158 - A target whose directory is absent from the worktree is skipped with a
159 warning, so sando can still build older shas from a config describing the
160 tip. If *no* target exists, the gate fails rather than reporting a pass over
161 zero suites.
162
163 ### What `migration_dry_run` restores
164
165 Every database with its own migrations needs its own check. multithreaded ships
166 its own migrations and applies them with `sqlx::migrate!()` at boot against its
167 own database, and sqlx checksums whole migration files, so an edited
168 already-applied migration fails to boot in prod rather than failing a dry run.
169
170 Each database gets a check in the daemon config, paired with its own dump in the
171 topology:
172
173 ```toml
174 # sando.toml (topology)
175 [[backup]]
176 name = "server"
177 source = "ssh://backup-puller@alpha-west-1:2200/makenotwork/latest.sql.gz"
178 local_path = "/srv/sando/backups/latest.sql.gz"
179
180 [[backup]]
181 name = "multithreaded"
182 source = "ssh://backup-puller@alpha-west-1:2200/multithreaded/latest.sql.gz"
183 local_path = "/srv/sando/backups/multithreaded-latest.sql.gz"
184 ```
185
186 ```toml
187 # sando-daemon.toml
188 [[migration_check]]
189 dir = "server/migrations"
190 backup = "server"
191
192 [[migration_check]]
193 dir = "multithreaded/migrations"
194 backup = "multithreaded"
195 scratch_db = "sando_scratch_mt"
196 owner_role = "multithreaded"
197 ```
198
199 Omitting either key defaults to one `server/migrations` check against one
200 `[backup]`, which parses as a single-entry list.
201
202 Notes on the semantics:
203
204 - **A check restores its own database's dump.** Restoring the server's dump under
205 another service's migrations would fail on the first migration for the least
206 interesting reason: a `_sqlx_migrations` table full of someone else's rows.
207 - `scratch_db` is what keeps checks from clobbering each other. The server check
208 leaves it unset, so it runs against `scratch_db_url` itself and leaves it in
209 migrated state for `cargo_test` to reuse; every other check names its own
210 database, which the daemon drops and recreates at the start of the check. Two
211 checks sharing one is refused at config load.
212 - `owner_role` defaults to `scratch_owner_role`. A dump carries
213 `ALTER ... OWNER TO <role>` for every object, and the role has to exist in the
214 scratch cluster before the restore, so a dump owned by anyone else needs this.
215 - Freshness and the fetch's plausibility floor are both per-dump. A fresh server
216 dump does not make a 45-day-old mt dump look current, and the server's size
217 does not set mt's floor (they differ by two orders of magnitude).
218 - All checks share one `gate_runs` row and one log file, sectioned by
219 `==== migration_check: <dir> ====` banners. The gate stops at the first red
220 check.
221 - A `[[migration_check]]` naming a `backup` the topology does not declare fails
222 at startup, not at the first promote.
223
224 ### What `code_smoke` builds first
225
226 Before it creates a database or boots anything, `code_smoke` compiles every
227 configured frontend:
228
229 ```toml
230 [[frontend_build]]
231 dir = "server/frontend"
232
233 [[frontend_build]]
234 dir = "multithreaded/frontend" # script = "build" by default
235 ```
236
237 These are npm projects whose compiled output the binary serves but whose failure
238 `cargo build` will not report. Both MNW crates compile TypeScript from a build
239 script that downgrades a `tsc` error to a `cargo::warning` and lets the Rust
240 build succeed against whatever `static/dist/` already holds, on purpose, so a
241 type error in a chat widget cannot stop the forum from compiling. Nothing else
242 downstream notices, and the deploy would rsync the previous build's bundle. This
243 is the one place that failure is fatal.
244
245 Semantics match `cargo_test`: `npm ci` first if `node_modules` is absent (usually
246 it is not, because the build script that produced the artifact already installed
247 it), stop at the first red project with the directory named, one deadline across
248 the gate, and a project absent from the worktree is skipped with a log line so
249 older shas still rebuild. Omitting the key entirely gates nothing, which is the
250 right default for a project with no frontend.
251
252 ### The lint and supply-chain gates
253
254 `clippy` and `fmt` run over the same `[[test_target]]` list as `cargo_test`,
255 with the same semantics: per-target log banners, stop at the first red target
256 with the crate named, one deadline across the whole gate.
257
258 `cargo_audit` and `cargo_deny` are **config-gated**: a target only qualifies
259 once it carries a `.cargo/audit.toml` or `deny.toml`. Both tools are only
260 meaningful against a triaged posture, and four crates in this repo fail
261 `cargo audit` purely for lack of a file recording which transitive advisories
262 have been reviewed and accepted. Running them everywhere would make the gate
263 permanently and uninformatively red, which teaches everyone to ignore it.
264 Dropping the config file into a crate is what opts it in.
265
266 ### Why `hardening_test` exists
267
268 `cargo_test` builds with `--features fast-tests`, which relaxes
269 `AUTH_RATE_LIMIT_BURST` (5 to 20), `SANDBOX_RATE_LIMIT_MS` (30s to 10ms), and
270 argon2 (46 MiB/t=2 down to 8 MiB/t=1) so the signup-heavy workflow suite
271 finishes in reasonable time. On top of that, the rate-limiting tests are
272 `#[cfg_attr(feature = "fast-tests", ignore)]`d, because a bucket refilling at
273 100/sec never depletes under parallel test threads.
274
275 So `cargo_test` covers none of the auth hardening it exists to protect.
276 `hardening_test` re-runs that suite with no features, single-threaded, against
277 production constants. It costs a second
278 compile of the server's test binary, since a different feature set is a
279 different cfg and shares no artifacts. It also fails closed if its name filter
280 matches zero tests, so renaming the suite cannot quietly turn the gate into a
281 green no-op.
282
283 ## API
284
285 | Method | Path | Body | Purpose |
286 |--------|------|------|---------|
287 | GET | `/state` || Tier list + current/previous version + last gate outcomes, plus `build` (latest build run: phase/result/failure_summary/elapsed_s, `null` until first `/rebuild`) so a poller sees in-flight/failed builds, not a frozen version |
288 | POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). Returns `{accepted, sha, run_id}`. |
289 | POST | `/intake` | `{staged, record}` | Accept an artifact built elsewhere and take it through the same host-tier gating a locally-built one gets. `staged` is a directory already under this app's `release_root/staging/`; `record` is the builder's `ArtifactRecord` verbatim. The bytes are proved against the record before anything else happens — a bundle that drifted in transit is refused with the offending file named. Returns `{accepted, run_id}`. |
290 | GET | `/runs/{id}` || Build-status of the run a `/rebuild` returned: `{run_id, sha, version, phase, result, failure_summary, gates[], started_at, finished_at}`. The pollable resource for a non-TUI driver; `/state` only reflects the last *successful* version. |
291 | GET | `/runs/{id}/wait` | `?timeout_ms=` | Long-poll: blocks until the run settles or `timeout_ms` (default 30s, cap 120s) elapses, then returns the same `RunView`. Fire `/rebuild` → block on `/wait`. |
292 | POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial`: the rollout landed, the tier cannot promote onward. |
293 | POST | `/rollback/{tier}` || Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. |
294 | POST | `/confirm/{tier}` || Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. |
295 | POST | `/backup/fetch` | `{force?, name?}` | Pull every configured prod dump, or just `name`. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one *of the same name*, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. Each dump is attempted even if another fails. |
296 | GET | `/events` || WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). |
297
298 ## TUI
299
300 `sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings:
301
302 | key | action |
303 |-----|--------|
304 | ↑/↓ or j/k | select tier |
305 | p | `POST /promote/<selected>` (no body; version defaults to predecessor's current) |
306 | R | `POST /rollback/<selected>` |
307 | b | `POST /backup/fetch` |
308 | c | `POST /confirm/<selected>` |
309 | r | refresh hint (poller is already every 2s) |
310 | q / Esc / Ctrl-C | quit |
311
312 Action results show up in the events log a moment later (the actions themselves emit events from the daemon side).
313
314 ## Hotfix flow
315
316 `POST /promote/{tier}` accepts:
317
318 - `hotfix: true` skips the `burn_in` gate on the predecessor tier only. All
319 other gates still apply, `manual_confirm` included: a hotfix still needs an
320 operator sign-off before it reaches production.
321 - `reset_burn_in: true` (default `false`) additionally nulls
322 `tier_state.burn_in_started_at` on the source tier, restarting the clock
323 for whatever else is still burning in there. Use this only when the hotfix
324 meaningfully changes the surface area under burn-in.
325
326 ## Shipping more than one product
327
328 One daemon, one database, one bind address, and beneath that N independent
329 pipelines, each with its own repo, tiers, nodes, gates, release root and version
330 history. `[app.<id>]` in `sando-daemon.toml` points at each product's own config;
331 a file with no `[app.*]` tables is read as the single app `mnw`.
332
333 The unprefixed routes address the default product, because `/promote/b` is what
334 the runbook says and what an operator types under pressure. Every product is also
335 at `/apps/<id>/...`, and `GET /apps` reports what is mounted.
336
337 **A product does not have to be one Sando builds.** `pom` is one that is not:
338 it runs on aarch64 and on x86_64, and Sando compiles on one configured host,
339 so it could never build half of a pom release without breaking its own
340 never-cross-compile rule. Bento builds it natively on both; Sando gates what
341 arrives and performs the advance. Such a product declares no `build_host` and no
342 `[repo]`. Absent is not "build anywhere", it is a statement that Sando does not
343 build this at all, and `/rebuild` refuses rather than choosing a machine.
344
345 ### Platforms, and why a bundle cannot land on the wrong box
346
347 One pom version is two bundles with two digests. Which one a node gets is not a
348 check before the deploy call, it is the only way to make the call:
349
350 ```rust
351 let placement = Placement::check(node, bundle, artifact_platform)?;
352 deploy_node(executor, placement, version, primary_bin).await
353 ```
354
355 `deploy_node` takes a `Placement`, and `Placement::check` is its only
356 constructor, so a mismatched deploy is not a bug to avoid but a value that
357 cannot be built. Both sides state a platform (`platform = "linux/aarch64"` on a
358 node, the artifact record's provenance for a bundle) and they must be equal.
359 Silence on one side is a refusal, not a pass; both silent is the single-platform
360 world MNW still lives in, and the moment either side starts stating, the other
361 has to as well.
362
363 Promote resolves per node before any node is touched, so a version missing its
364 x86_64 half fails whole rather than halfway down a rollout. A sibling bundle only
365 qualifies if its own run settled green: each architecture stands on its own
366 intake and its own gate run, because the source tier's evidence says nothing
367 about bytes it never saw.
368
369 ### Which gates go where
370
371 Evidence *about the artifact* (`cargo_test`, `clippy`, `fmt`, the audits)
372 belongs to the builder. Evidence *about the artifact in an environment*
373 (`migration_dry_run`, `boot_smoke`, `node_health`, `burn_in`, `manual_confirm`)
374 is Sando's, which also keeps production dumps on the machine that already has
375 them instead of handing them to build hosts.
376
377 An accepted artifact has no worktree, so a source-reading gate configured on its
378 tier refuses rather than resolving against nothing and reporting green.
379 `migration_dry_run` resolves its migrations from the bundle first and the
380 worktree second, which is why MNW stages `server/migrations` and
381 `multithreaded/migrations` into the bundle: inside the digest, the gate proves
382 something about the bytes that ship rather than about a checkout sitting beside
383 them.
384
385 ## v0 limitations
386
387 - `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The
388 gate drops every non-system schema on every run; do not point this at
389 anything that matters. A check with its own `scratch_db` gets that whole
390 database dropped and recreated instead.
391
392 ## License
393
394 MIT. The surrounding MNW monorepo is PolyForm-Noncommercial; sando is
395 deliberately MIT'd because it's deploy infra, not the product.
396