# sando Home-rolled CI/CD controller for the MNW server. Axum daemon (`sandod`) + ratatui TUI (`sando`). Gates a tiered deploy flow: ``` git push mm -> MakeMachine (build + tests + migration dry-run + boot smoke) -> A (testnot.work) -> B (prod-1) -> C (prod-2) ``` Each tier's progression gates are declared in `sando.toml`. Tiers and nodes live in the TOML, not in code; adding a node or a new tier is a config edit. ## Crates | Path | Binary | Role | |------|--------|------| | `daemon/` | `sandod` | Axum daemon. Runs on the MakeMachine. Owns SQLite state, the bare git repo, and all build/gate/deploy logic. | | `tui/` | `sando` | ratatui front-end. Runs on the laptop. Talks to `sandod` over the tailnet. | ## Quickstart: localhost dev loop The MakeMachine hardware does not exist yet, so v0 runs entirely on a single host. Bare repo, releases dir, "remote" A node — everything is a local directory. ```bash # 1. Build both binaries. cd MNW/sando/daemon && cargo build cd ../tui && cargo build # 2. Create a workspace + config. mkdir -p /tmp/sando-dev cat > /tmp/sando-dev/daemon.toml < /tmp/sando-dev/sando.toml </`, then runs the host tier's gates. On green the bundle is published content-addressed at `releases//` (the sha256 of its `MANIFEST` rather than the version; see `daemon/src/bundle.rs`), `current` is swapped to it, and the host tier's `tier_state` advances. Promote with: ```bash curl -X POST http://127.0.0.1:7766/promote/a \ -H 'Content-Type: application/json' \ -d '{"version":"0.8.2"}' ``` ## Gates Build-time gates run on the host tier, once per build, against the worktree. Promote-time gates run against a tier's deployed nodes or its operator. | Kind | When | What it proves | |------|------|----------------| | `code_smoke` | build | Compiles every `[[frontend_build]]` (see below), then boots the fresh binary on a throwaway DB it migrates from scratch and seeds, then probes `/health`. Runs first: green here isolates a later red as an environment problem, not a code one. | | `fmt` | build | `cargo fmt --check` over every `[[test_target]]`. No compilation, so it fails fast. | | `cargo_test` | build | Every configured `[[test_target]]` crate's suite, in order (see below). | | `hardening_test` | build | What `cargo_test` structurally cannot reach (see below). | | `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. | | `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. | | `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. | | `migration_dry_run` | build | Every configured `[[migration_check]]` database's migrations apply cleanly to a restored production dump of *that* database. Blocks if a check's newest fetched dump is older than `backup_max_age_hours` (default 48); a stale dump proves nothing about today's schema. | | `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. | | `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. | | `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. | | `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. | A tier's gates guard promotion **out** of it. So the gate list that stands between staging and production is tier `a`'s, not tier `b`'s. Sign-off for a prod ship is `POST /confirm/a`, run after the version lands on `a` (the confirmation must be fresher than that landing) and before `POST /promote/b`. ### What `cargo_test` runs Targets are configured in the daemon config: ```toml [[test_target]] dir = "server" features = ["fast-tests"] scratch_db = true # export DATABASE_URL / TEST_DATABASE_URL [[test_target]] dir = "shared/tagtree" # no features, no DB [[test_target]] dir = "shared/ops-exec" all_features = true # mutually exclusive with `features` ``` Omitting the key entirely defaults to one `server` target with `fast-tests`, against the scratch DB. Notes on the semantics: - `scratch_db` is opt-in per target. Setting `DATABASE_URL` takes sqlx **out** of offline mode, so a crate that ships `.sqlx` query data would try to type-check against a database holding none of its tables. - All targets share one `gate_runs` row and one log file, sectioned by `==== test_target: ====` banners. The gate stops at the first red target, and the failure summary names the crate. - `gate_timeout_secs` bounds the whole gate, not each target. - A target whose directory is absent from the worktree is skipped with a warning, so sando can still build older shas from a config describing the tip. If *no* target exists, the gate fails rather than reporting a pass over zero suites. ### What `migration_dry_run` restores Every database with its own migrations needs its own check. multithreaded ships its own migrations and applies them with `sqlx::migrate!()` at boot against its own database, and sqlx checksums whole migration files, so an edited already-applied migration fails to boot in prod rather than failing a dry run. Each database gets a check in the daemon config, paired with its own dump in the topology: ```toml # sando.toml (topology) [[backup]] name = "server" source = "ssh://backup-puller@alpha-west-1:2200/makenotwork/latest.sql.gz" local_path = "/srv/sando/backups/latest.sql.gz" [[backup]] name = "multithreaded" source = "ssh://backup-puller@alpha-west-1:2200/multithreaded/latest.sql.gz" local_path = "/srv/sando/backups/multithreaded-latest.sql.gz" ``` ```toml # sando-daemon.toml [[migration_check]] dir = "server/migrations" backup = "server" [[migration_check]] dir = "multithreaded/migrations" backup = "multithreaded" scratch_db = "sando_scratch_mt" owner_role = "multithreaded" ``` Omitting either key defaults to one `server/migrations` check against one `[backup]`, which parses as a single-entry list. Notes on the semantics: - **A check restores its own database's dump.** Restoring the server's dump under another service's migrations would fail on the first migration for the least interesting reason: a `_sqlx_migrations` table full of someone else's rows. - `scratch_db` is what keeps checks from clobbering each other. The server check leaves it unset, so it runs against `scratch_db_url` itself and leaves it in migrated state for `cargo_test` to reuse; every other check names its own database, which the daemon drops and recreates at the start of the check. Two checks sharing one is refused at config load. - `owner_role` defaults to `scratch_owner_role`. A dump carries `ALTER ... OWNER TO ` for every object, and the role has to exist in the scratch cluster before the restore, so a dump owned by anyone else needs this. - Freshness and the fetch's plausibility floor are both per-dump. A fresh server dump does not make a 45-day-old mt dump look current, and the server's size does not set mt's floor (they differ by two orders of magnitude). - All checks share one `gate_runs` row and one log file, sectioned by `==== migration_check: ====` banners. The gate stops at the first red check. - A `[[migration_check]]` naming a `backup` the topology does not declare fails at startup, not at the first promote. ### What `code_smoke` builds first Before it creates a database or boots anything, `code_smoke` compiles every configured frontend: ```toml [[frontend_build]] dir = "server/frontend" [[frontend_build]] dir = "multithreaded/frontend" # script = "build" by default ``` These are npm projects whose compiled output the binary serves but whose failure `cargo build` will not report. Both MNW crates compile TypeScript from a build script that downgrades a `tsc` error to a `cargo::warning` and lets the Rust build succeed against whatever `static/dist/` already holds, on purpose, so a type error in a chat widget cannot stop the forum from compiling. Nothing else downstream notices, and the deploy would rsync the previous build's bundle. This is the one place that failure is fatal. Semantics match `cargo_test`: `npm ci` first if `node_modules` is absent (usually it is not, because the build script that produced the artifact already installed it), stop at the first red project with the directory named, one deadline across the gate, and a project absent from the worktree is skipped with a log line so older shas still rebuild. Omitting the key entirely gates nothing, which is the right default for a project with no frontend. ### The lint and supply-chain gates `clippy` and `fmt` run over the same `[[test_target]]` list as `cargo_test`, with the same semantics: per-target log banners, stop at the first red target with the crate named, one deadline across the whole gate. `cargo_audit` and `cargo_deny` are **config-gated**: a target only qualifies once it carries a `.cargo/audit.toml` or `deny.toml`. Both tools are only meaningful against a triaged posture, and four crates in this repo fail `cargo audit` purely for lack of a file recording which transitive advisories have been reviewed and accepted. Running them everywhere would make the gate permanently and uninformatively red, which teaches everyone to ignore it. Dropping the config file into a crate is what opts it in. ### Why `hardening_test` exists `cargo_test` builds with `--features fast-tests`, which relaxes `AUTH_RATE_LIMIT_BURST` (5 to 20), `SANDBOX_RATE_LIMIT_MS` (30s to 10ms), and argon2 (46 MiB/t=2 down to 8 MiB/t=1) so the signup-heavy workflow suite finishes in reasonable time. On top of that, the rate-limiting tests are `#[cfg_attr(feature = "fast-tests", ignore)]`d, because a bucket refilling at 100/sec never depletes under parallel test threads. So `cargo_test` covers none of the auth hardening it exists to protect. `hardening_test` re-runs that suite with no features, single-threaded, against production constants. It costs a second compile of the server's test binary, since a different feature set is a different cfg and shares no artifacts. It also fails closed if its name filter matches zero tests, so renaming the suite cannot quietly turn the gate into a green no-op. ## API | Method | Path | Body | Purpose | |--------|------|------|---------| | GET | `/state` | — | Tier list + current/previous version + last gate outcomes, plus `build` (latest build run: phase/result/failure_summary/elapsed_s, `null` until first `/rebuild`) so a poller sees in-flight/failed builds, not a frozen version | | POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). Returns `{accepted, sha, run_id}`. | | POST | `/intake` | `{staged, record}` | Accept an artifact built elsewhere and take it through the same host-tier gating a locally-built one gets. `staged` is a directory already under this app's `release_root/staging/`; `record` is the builder's `ArtifactRecord` verbatim. The bytes are proved against the record before anything else happens — a bundle that drifted in transit is refused with the offending file named. Returns `{accepted, run_id}`. | | GET | `/runs/{id}` | — | Build-status of the run a `/rebuild` returned: `{run_id, sha, version, phase, result, failure_summary, gates[], started_at, finished_at}`. The pollable resource for a non-TUI driver; `/state` only reflects the last *successful* version. | | GET | `/runs/{id}/wait` | `?timeout_ms=` | Long-poll: blocks until the run settles or `timeout_ms` (default 30s, cap 120s) elapses, then returns the same `RunView`. Fire `/rebuild` → block on `/wait`. | | POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial`: the rollout landed, the tier cannot promote onward. | | POST | `/rollback/{tier}` | — | Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. | | POST | `/confirm/{tier}` | — | Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. | | POST | `/backup/fetch` | `{force?, name?}` | Pull every configured prod dump, or just `name`. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one *of the same name*, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. Each dump is attempted even if another fails. | | GET | `/events` | — | WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). | ## TUI `sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings: | key | action | |-----|--------| | ↑/↓ or j/k | select tier | | p | `POST /promote/` (no body; version defaults to predecessor's current) | | R | `POST /rollback/` | | b | `POST /backup/fetch` | | c | `POST /confirm/` | | r | refresh hint (poller is already every 2s) | | q / Esc / Ctrl-C | quit | Action results show up in the events log a moment later (the actions themselves emit events from the daemon side). ## Hotfix flow `POST /promote/{tier}` accepts: - `hotfix: true` skips the `burn_in` gate on the predecessor tier only. All other gates still apply, `manual_confirm` included: a hotfix still needs an operator sign-off before it reaches production. - `reset_burn_in: true` (default `false`) additionally nulls `tier_state.burn_in_started_at` on the source tier, restarting the clock for whatever else is still burning in there. Use this only when the hotfix meaningfully changes the surface area under burn-in. ## Shipping more than one product One daemon, one database, one bind address, and beneath that N independent pipelines, each with its own repo, tiers, nodes, gates, release root and version history. `[app.]` in `sando-daemon.toml` points at each product's own config; a file with no `[app.*]` tables is read as the single app `mnw`. The unprefixed routes address the default product, because `/promote/b` is what the runbook says and what an operator types under pressure. Every product is also at `/apps//...`, and `GET /apps` reports what is mounted. **A product does not have to be one Sando builds.** `pom` is one that is not: it runs on aarch64 and on x86_64, and Sando compiles on one configured host, so it could never build half of a pom release without breaking its own never-cross-compile rule. Bento builds it natively on both; Sando gates what arrives and performs the advance. Such a product declares no `build_host` and no `[repo]`. Absent is not "build anywhere", it is a statement that Sando does not build this at all, and `/rebuild` refuses rather than choosing a machine. ### Platforms, and why a bundle cannot land on the wrong box One pom version is two bundles with two digests. Which one a node gets is not a check before the deploy call, it is the only way to make the call: ```rust let placement = Placement::check(node, bundle, artifact_platform)?; deploy_node(executor, placement, version, primary_bin).await ``` `deploy_node` takes a `Placement`, and `Placement::check` is its only constructor, so a mismatched deploy is not a bug to avoid but a value that cannot be built. Both sides state a platform (`platform = "linux/aarch64"` on a node, the artifact record's provenance for a bundle) and they must be equal. Silence on one side is a refusal, not a pass; both silent is the single-platform world MNW still lives in, and the moment either side starts stating, the other has to as well. Promote resolves per node before any node is touched, so a version missing its x86_64 half fails whole rather than halfway down a rollout. A sibling bundle only qualifies if its own run settled green: each architecture stands on its own intake and its own gate run, because the source tier's evidence says nothing about bytes it never saw. ### Which gates go where Evidence *about the artifact* (`cargo_test`, `clippy`, `fmt`, the audits) belongs to the builder. Evidence *about the artifact in an environment* (`migration_dry_run`, `boot_smoke`, `node_health`, `burn_in`, `manual_confirm`) is Sando's, which also keeps production dumps on the machine that already has them instead of handing them to build hosts. An accepted artifact has no worktree, so a source-reading gate configured on its tier refuses rather than resolving against nothing and reporting green. `migration_dry_run` resolves its migrations from the bundle first and the worktree second, which is why MNW stages `server/migrations` and `multithreaded/migrations` into the bundle: inside the digest, the gate proves something about the bytes that ship rather than about a checkout sitting beside them. ## v0 limitations - `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The gate drops every non-system schema on every run; do not point this at anything that matters. A check with its own `scratch_db` gets that whole database dropped and recreated instead. ## License MIT. The surrounding MNW monorepo is PolyForm-Noncommercial; sando is deliberately MIT'd because it's deploy infra, not the product.