| 1 |
# sando |
| 2 |
|
| 3 |
Home-rolled CI/CD controller for the MNW server. Axum daemon (`sandod`) + |
| 4 |
ratatui TUI (`sando`). Gates a tiered deploy flow: |
| 5 |
|
| 6 |
``` |
| 7 |
git push mm -> MakeMachine (build + tests + migration dry-run + boot smoke) |
| 8 |
-> A (testnot.work) |
| 9 |
-> B (prod-1) |
| 10 |
-> C (prod-2) |
| 11 |
``` |
| 12 |
|
| 13 |
Each tier's progression gates are declared in `sando.toml`. Tiers and nodes |
| 14 |
live in the TOML, not in code — adding a node or a new tier is a config edit. |
| 15 |
|
| 16 |
## Crates |
| 17 |
|
| 18 |
|
| 19 |
|
| 20 |
| `daemon/` | `sandod` | Axum daemon. Runs on the MakeMachine. Owns SQLite state, the bare git repo, and all build/gate/deploy logic. | |
| 21 |
| `tui/` | `sando` | ratatui front-end. Runs on the laptop. Talks to `sandod` over the tailnet. | |
| 22 |
|
| 23 |
## Quickstart: localhost dev loop |
| 24 |
|
| 25 |
The MakeMachine hardware does not exist yet, so v0 runs entirely on a single |
| 26 |
host. Bare repo, releases dir, "remote" A node — everything is a local |
| 27 |
directory. |
| 28 |
|
| 29 |
```bash |
| 30 |
# 1. Build both binaries. |
| 31 |
cd MNW/sando/daemon && cargo build |
| 32 |
cd ../tui && cargo build |
| 33 |
|
| 34 |
# 2. Create a workspace + config. |
| 35 |
mkdir -p /tmp/sando-dev |
| 36 |
cat > /tmp/sando-dev/daemon.toml <<EOF |
| 37 |
listen = "127.0.0.1:7766" |
| 38 |
db_path = "/tmp/sando-dev/sando.db" |
| 39 |
topology_path = "/tmp/sando-dev/sando.toml" |
| 40 |
workdir = "/tmp/sando-dev/work" |
| 41 |
release_root = "/tmp/sando-dev/releases" |
| 42 |
# scratch_db_url = "postgres://you@127.0.0.1/sando_scratch" |
| 43 |
EOF |
| 44 |
|
| 45 |
cat > /tmp/sando-dev/sando.toml <<EOF |
| 46 |
[repo] |
| 47 |
bare_path = "/tmp/sando-dev/mnw.git" |
| 48 |
branch = "main" |
| 49 |
[backup] |
| 50 |
source = "file:///tmp/sando-dev/fake-backup.sql" |
| 51 |
local_path = "/tmp/sando-dev/backup.sql" |
| 52 |
|
| 53 |
[[tier]] |
| 54 |
name = "mm" |
| 55 |
provisioned = true |
| 56 |
gates = [ |
| 57 |
{ kind = "cargo_test" }, |
| 58 |
{ kind = "migration_dry_run" }, |
| 59 |
{ kind = "boot_smoke" }, |
| 60 |
] |
| 61 |
|
| 62 |
[[tier]] |
| 63 |
name = "a" |
| 64 |
provisioned = true |
| 65 |
canary = "sequential" |
| 66 |
gates = [ |
| 67 |
{ kind = "boot_smoke" }, |
| 68 |
{ kind = "manual_confirm" }, |
| 69 |
] |
| 70 |
[[tier.node]] |
| 71 |
name = "a-local" |
| 72 |
ssh_target = "local" |
| 73 |
release_root = "/tmp/sando-dev/a-node" |
| 74 |
EOF |
| 75 |
|
| 76 |
# 3. Run the daemon. |
| 77 |
SANDO_CONFIG=/tmp/sando-dev/daemon.toml \ |
| 78 |
./MNW/sando/daemon/target/debug/sandod |
| 79 |
|
| 80 |
# 4. In another shell: point a clone at the bare repo and push. |
| 81 |
git clone /tmp/sando-dev/mnw.git /tmp/sando-dev/checkout |
| 82 |
# ... add a `server/Cargo.toml` + source so the build can run ... |
| 83 |
cd /tmp/sando-dev/checkout && git push origin main |
| 84 |
|
| 85 |
# 5. Watch the TUI. |
| 86 |
SANDO_DAEMON=http://127.0.0.1:7766 ./MNW/sando/tui/target/debug/sando |
| 87 |
``` |
| 88 |
|
| 89 |
When you push, the bare repo's `post-receive` hook (installed automatically |
| 90 |
by `sandod` on startup) calls `POST /rebuild`. The daemon checks out the |
| 91 |
sha, runs `cargo build --release` against `server/`, stages the binary in |
| 92 |
`releases/<version>/server`, then runs the MM tier's gates. On green, MM's |
| 93 |
`tier_state` advances. Promote with: |
| 94 |
|
| 95 |
```bash |
| 96 |
curl -X POST http://127.0.0.1:7766/promote/a \ |
| 97 |
-H 'Content-Type: application/json' \ |
| 98 |
-d '{"version":"0.8.2"}' |
| 99 |
``` |
| 100 |
|
| 101 |
## Gates |
| 102 |
|
| 103 |
Build-time gates run on the host tier, once per build, against the worktree. |
| 104 |
Promote-time gates run against a tier's deployed nodes or its operator. |
| 105 |
|
| 106 |
|
| 107 |
|
| 108 |
| `code_smoke` | build | Compiles every `[[frontend_build]]` (see below), then boots the fresh binary on a throwaway DB it migrates from scratch and seeds, then probes `/health`. Runs first: green here isolates a later red as an environment problem, not a code one. | |
| 109 |
| `fmt` | build | `cargo fmt --check` over every `[[test_target]]`. No compilation, so it fails fast. | |
| 110 |
| `cargo_test` | build | Every configured `[[test_target]]` crate's suite, in order (see below). | |
| 111 |
| `hardening_test` | build | What `cargo_test` structurally cannot reach (see below). | |
| 112 |
| `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. | |
| 113 |
| `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. | |
| 114 |
| `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. | |
| 115 |
| `migration_dry_run` | build | Migrations apply cleanly to a restored production dump. Blocks if the newest fetched dump is older than `backup_max_age_hours` (default 48) — a stale dump proves nothing about today's schema. | |
| 116 |
| `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. | |
| 117 |
| `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. | |
| 118 |
| `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. | |
| 119 |
| `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. | |
| 120 |
|
| 121 |
A tier's gates guard promotion **out** of it. So the gate list that stands |
| 122 |
between staging and production is tier `a`'s, not tier `b`'s — sign-off for a |
| 123 |
prod ship is `POST /confirm/a`, run after the version lands on `a` (the |
| 124 |
confirmation must be fresher than that landing) and before `POST /promote/b`. |
| 125 |
|
| 126 |
### What `cargo_test` runs |
| 127 |
|
| 128 |
The gate used to be hardcoded to `worktree/server`, so every other crate in the |
| 129 |
repo shipped ungated. `mnw-cli` was the sharp edge: it is built as a companion |
| 130 |
and installed onto prod-1 in the same promote that ships the server, with no |
| 131 |
test ever run against it. |
| 132 |
|
| 133 |
Targets are now configured in the daemon config: |
| 134 |
|
| 135 |
```toml |
| 136 |
[[test_target]] |
| 137 |
dir = "server" |
| 138 |
features = ["fast-tests"] |
| 139 |
scratch_db = true # export DATABASE_URL / TEST_DATABASE_URL |
| 140 |
|
| 141 |
[[test_target]] |
| 142 |
dir = "shared/tagtree" # no features, no DB |
| 143 |
|
| 144 |
[[test_target]] |
| 145 |
dir = "shared/ops-exec" |
| 146 |
all_features = true # mutually exclusive with `features` |
| 147 |
``` |
| 148 |
|
| 149 |
Omitting the key entirely keeps the historical behavior: one `server` target |
| 150 |
with `fast-tests`, against the scratch DB. |
| 151 |
|
| 152 |
Notes on the semantics: |
| 153 |
|
| 154 |
- `scratch_db` is opt-in per target. Setting `DATABASE_URL` takes sqlx **out** |
| 155 |
of offline mode, so a crate that ships `.sqlx` query data would try to |
| 156 |
type-check against a database holding none of its tables. |
| 157 |
- All targets share one `gate_runs` row and one log file, sectioned by |
| 158 |
`==== test_target: <dir> ====` banners. The gate stops at the first red |
| 159 |
target, and the failure summary names the crate. |
| 160 |
- `gate_timeout_secs` bounds the whole gate, not each target. |
| 161 |
- A target whose directory is absent from the worktree is skipped with a |
| 162 |
warning, so sando can still build older shas from a config describing the |
| 163 |
tip. If *no* target exists, the gate fails rather than reporting a pass over |
| 164 |
zero suites. |
| 165 |
|
| 166 |
### What `code_smoke` builds first |
| 167 |
|
| 168 |
Before it creates a database or boots anything, `code_smoke` compiles every |
| 169 |
configured frontend: |
| 170 |
|
| 171 |
```toml |
| 172 |
[[frontend_build]] |
| 173 |
dir = "server/frontend" |
| 174 |
|
| 175 |
[[frontend_build]] |
| 176 |
dir = "multithreaded/frontend" # script = "build" by default |
| 177 |
``` |
| 178 |
|
| 179 |
These are npm projects whose compiled output the binary serves but whose failure |
| 180 |
`cargo build` will not report. Both MNW crates compile TypeScript from a build |
| 181 |
script that downgrades a `tsc` error to a `cargo::warning` and lets the Rust |
| 182 |
build succeed against whatever `static/dist/` already holds, on purpose, so a |
| 183 |
type error in a chat widget cannot stop the forum from compiling. The cost is |
| 184 |
that nothing downstream noticed either, and the deploy rsynced the previous |
| 185 |
build's bundle. This is the one place that failure is fatal. |
| 186 |
|
| 187 |
Semantics match `cargo_test`: `npm ci` first if `node_modules` is absent (usually |
| 188 |
it is not, because the build script that produced the artifact already installed |
| 189 |
it), stop at the first red project with the directory named, one deadline across |
| 190 |
the gate, and a project absent from the worktree is skipped with a log line so |
| 191 |
older shas still rebuild. Omitting the key entirely gates nothing, which is the |
| 192 |
right default for a project with no frontend. |
| 193 |
|
| 194 |
### The lint and supply-chain gates |
| 195 |
|
| 196 |
`clippy` and `fmt` run over the same `[[test_target]]` list as `cargo_test`, |
| 197 |
with the same semantics: per-target log banners, stop at the first red target |
| 198 |
with the crate named, one deadline across the whole gate. |
| 199 |
|
| 200 |
`cargo_audit` and `cargo_deny` are **config-gated**: a target only qualifies |
| 201 |
once it carries a `.cargo/audit.toml` or `deny.toml`. Both tools are only |
| 202 |
meaningful against a triaged posture, and four crates in this repo fail |
| 203 |
`cargo audit` purely for lack of a file recording which transitive advisories |
| 204 |
have been reviewed and accepted. Running them everywhere would make the gate |
| 205 |
permanently and uninformatively red, which teaches everyone to ignore it. |
| 206 |
Dropping the config file into a crate is what opts it in. |
| 207 |
|
| 208 |
Before these existed, `-D warnings` was enforced in exactly one place |
| 209 |
(`server/deploy/run-ci.sh`, which died with the astra pipeline) and |
| 210 |
`cargo fmt --check` nowhere at all. |
| 211 |
|
| 212 |
### Why `hardening_test` exists |
| 213 |
|
| 214 |
`cargo_test` builds with `--features fast-tests`, which relaxes |
| 215 |
`AUTH_RATE_LIMIT_BURST` (5 to 20), `SANDBOX_RATE_LIMIT_MS` (30s to 10ms), and |
| 216 |
argon2 (46 MiB/t=2 down to 8 MiB/t=1) so the signup-heavy workflow suite |
| 217 |
finishes in reasonable time. On top of that, the rate-limiting tests are |
| 218 |
`#[cfg_attr(feature = "fast-tests", ignore)]`d, because a bucket refilling at |
| 219 |
100/sec never depletes under parallel test threads. |
| 220 |
|
| 221 |
The net effect was that Sando's only code gate skipped every test of the auth |
| 222 |
hardening it exists to protect. `hardening_test` re-runs that suite with no |
| 223 |
features, single-threaded, against production constants. It costs a second |
| 224 |
compile of the server's test binary, since a different feature set is a |
| 225 |
different cfg and shares no artifacts. It also fails closed if its name filter |
| 226 |
matches zero tests, so renaming the suite cannot quietly turn the gate into a |
| 227 |
green no-op. |
| 228 |
|
| 229 |
## API |
| 230 |
|
| 231 |
|
| 232 |
|
| 233 |
| GET | `/state` | — | Tier list + current/previous version + last gate outcomes, plus `build` (latest build run: phase/result/failure_summary/elapsed_s, `null` until first `/rebuild`) so a poller sees in-flight/failed builds, not a frozen version | |
| 234 |
| POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). Returns `{accepted, sha, run_id}`. | |
| 235 |
| GET | `/runs/{id}` | — | Build-status of the run a `/rebuild` returned: `{run_id, sha, version, phase, result, failure_summary, gates[], started_at, finished_at}`. The pollable resource for a non-TUI driver — `/state` only reflects the last *successful* version. | |
| 236 |
| GET | `/runs/{id}/wait` | `?timeout_ms=` | Long-poll: blocks until the run settles or `timeout_ms` (default 30s, cap 120s) elapses, then returns the same `RunView`. Fire `/rebuild` → block on `/wait`. | |
| 237 |
| POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial` — the rollout landed, the tier cannot promote onward. | |
| 238 |
| POST | `/rollback/{tier}` | — | Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. | |
| 239 |
| POST | `/confirm/{tier}` | — | Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. | |
| 240 |
| POST | `/backup/fetch` | `{force?}` | Pull the prod backup. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. | |
| 241 |
| GET | `/events` | — | WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). | |
| 242 |
|
| 243 |
## TUI |
| 244 |
|
| 245 |
`sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings: |
| 246 |
|
| 247 |
|
| 248 |
|
| 249 |
| ↑/↓ or j/k | select tier | |
| 250 |
| p | `POST /promote/<selected>` (no body — version defaults to predecessor's current) | |
| 251 |
| R | `POST /rollback/<selected>` | |
| 252 |
| b | `POST /backup/fetch` | |
| 253 |
| c | `POST /confirm/<selected>` | |
| 254 |
| r | refresh hint (poller is already every 2s) | |
| 255 |
| q / Esc / Ctrl-C | quit | |
| 256 |
|
| 257 |
Action results show up in the events log a moment later (the actions themselves emit events from the daemon side). |
| 258 |
|
| 259 |
## Hotfix flow |
| 260 |
|
| 261 |
`POST /promote/{tier}` accepts: |
| 262 |
|
| 263 |
- `hotfix: true` — skips the `burn_in` gate on the predecessor tier only. All |
| 264 |
other gates still apply, `manual_confirm` included: a hotfix still needs an |
| 265 |
operator sign-off before it reaches production. |
| 266 |
- `reset_burn_in: true` (default `false`) — additionally nulls |
| 267 |
`tier_state.burn_in_started_at` on the source tier, restarting the clock |
| 268 |
for whatever else is still burning in there. Use this only when the hotfix |
| 269 |
meaningfully changes the surface area under burn-in. |
| 270 |
|
| 271 |
## v0 limitations |
| 272 |
|
| 273 |
- `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The |
| 274 |
gate drops every non-system schema on every run; do not point this at |
| 275 |
anything that matters. |
| 276 |
|
| 277 |
## License |
| 278 |
|
| 279 |
MIT. The surrounding MNW monorepo is PolyForm-Noncommercial — sando is |
| 280 |
deliberately MIT'd because it's deploy infra, not the product. |
| 281 |
|