# sando

Home-rolled CI/CD controller for the MNW server. Axum daemon (`sandod`) +
ratatui TUI (`sando`). Gates a tiered deploy flow:

```
git push mm  ->  MakeMachine (build + tests + migration dry-run + boot smoke)
              ->  A (testnot.work)
              ->  B (prod-1)
              ->  C (prod-2)
```

Each tier's progression gates are declared in `sando.toml`. Tiers and nodes
live in the TOML, not in code — adding a node or a new tier is a config edit.

## Crates

| Path | Binary | Role |
|------|--------|------|
| `daemon/` | `sandod` | Axum daemon. Runs on the MakeMachine. Owns SQLite state, the bare git repo, and all build/gate/deploy logic. |
| `tui/` | `sando` | ratatui front-end. Runs on the laptop. Talks to `sandod` over the tailnet. |

## Quickstart: localhost dev loop

The MakeMachine hardware does not exist yet, so v0 runs entirely on a single
host. Bare repo, releases dir, "remote" A node — everything is a local
directory.

```bash
# 1. Build both binaries.
cd MNW/sando/daemon && cargo build
cd ../tui            && cargo build

# 2. Create a workspace + config.
mkdir -p /tmp/sando-dev
cat > /tmp/sando-dev/daemon.toml <<EOF
listen = "127.0.0.1:7766"
db_path = "/tmp/sando-dev/sando.db"
topology_path = "/tmp/sando-dev/sando.toml"
workdir = "/tmp/sando-dev/work"
release_root = "/tmp/sando-dev/releases"
# scratch_db_url = "postgres://you@127.0.0.1/sando_scratch"
EOF

cat > /tmp/sando-dev/sando.toml <<EOF
[repo]
bare_path = "/tmp/sando-dev/mnw.git"
branch = "main"
[backup]
source = "file:///tmp/sando-dev/fake-backup.sql"
local_path = "/tmp/sando-dev/backup.sql"

[[tier]]
name = "mm"
provisioned = true
gates = [
    { kind = "cargo_test" },
    { kind = "migration_dry_run" },
    { kind = "boot_smoke" },
]

[[tier]]
name = "a"
provisioned = true
canary = "sequential"
gates = [
    { kind = "boot_smoke" },
    { kind = "manual_confirm" },
]
[[tier.node]]
name = "a-local"
ssh_target = "local"
release_root = "/tmp/sando-dev/a-node"
EOF

# 3. Run the daemon.
SANDO_CONFIG=/tmp/sando-dev/daemon.toml \
    ./MNW/sando/daemon/target/debug/sandod

# 4. In another shell: point a clone at the bare repo and push.
git clone /tmp/sando-dev/mnw.git /tmp/sando-dev/checkout
# ... add a `server/Cargo.toml` + source so the build can run ...
cd /tmp/sando-dev/checkout && git push origin main

# 5. Watch the TUI.
SANDO_DAEMON=http://127.0.0.1:7766 ./MNW/sando/tui/target/debug/sando
```

When you push, the bare repo's `post-receive` hook (installed automatically
by `sandod` on startup) calls `POST /rebuild`. The daemon checks out the
sha, runs `cargo build --release` against `server/`, stages the binary in
`releases/<version>/server`, then runs the MM tier's gates. On green, MM's
`tier_state` advances. Promote with:

```bash
curl -X POST http://127.0.0.1:7766/promote/a \
     -H 'Content-Type: application/json' \
     -d '{"version":"0.8.2"}'
```

## Gates

Build-time gates run on the host tier, once per build, against the worktree.
Promote-time gates run against a tier's deployed nodes or its operator.

| Kind | When | What it proves |
|------|------|----------------|
| `code_smoke` | build | Compiles every `[[frontend_build]]` (see below), then boots the fresh binary on a throwaway DB it migrates from scratch and seeds, then probes `/health`. Runs first: green here isolates a later red as an environment problem, not a code one. |
| `fmt` | build | `cargo fmt --check` over every `[[test_target]]`. No compilation, so it fails fast. |
| `cargo_test` | build | Every configured `[[test_target]]` crate's suite, in order (see below). |
| `hardening_test` | build | What `cargo_test` structurally cannot reach (see below). |
| `clippy` | build | `cargo clippy --all-targets -- -D warnings` over every `[[test_target]]`. |
| `cargo_audit` | build | `cargo audit` in each `[[test_target]]` carrying a `.cargo/audit.toml`. |
| `cargo_deny` | build | `cargo deny check` in each `[[test_target]]` carrying a `deny.toml`. |
| `migration_dry_run` | build | Migrations apply cleanly to a restored production dump. Blocks if the newest fetched dump is older than `backup_max_age_hours` (default 48) — a stale dump proves nothing about today's schema. |
| `boot_smoke` | build | The staged artifact boots in minimal no-DB mode on the build host. |
| `node_health` | post-deploy | Each deployed node's unit is active (and serves 2xx if `health_url` is set). Recorded at the end of a promote as the evidence the next promote checks. |
| `burn_in` | promote | The tier has held its current version for N hours. Evaluated live against the clock. |
| `manual_confirm` | promote | An operator signed off, at or after this version landed on the tier. |

A tier's gates guard promotion **out** of it. So the gate list that stands
between staging and production is tier `a`'s, not tier `b`'s — sign-off for a
prod ship is `POST /confirm/a`, run after the version lands on `a` (the
confirmation must be fresher than that landing) and before `POST /promote/b`.

### What `cargo_test` runs

The gate used to be hardcoded to `worktree/server`, so every other crate in the
repo shipped ungated. `mnw-cli` was the sharp edge: it is built as a companion
and installed onto prod-1 in the same promote that ships the server, with no
test ever run against it.

Targets are now configured in the daemon config:

```toml
[[test_target]]
dir = "server"
features = ["fast-tests"]
scratch_db = true       # export DATABASE_URL / TEST_DATABASE_URL

[[test_target]]
dir = "shared/tagtree"  # no features, no DB

[[test_target]]
dir = "shared/ops-exec"
all_features = true     # mutually exclusive with `features`
```

Omitting the key entirely keeps the historical behavior: one `server` target
with `fast-tests`, against the scratch DB.

Notes on the semantics:

- `scratch_db` is opt-in per target. Setting `DATABASE_URL` takes sqlx **out**
  of offline mode, so a crate that ships `.sqlx` query data would try to
  type-check against a database holding none of its tables.
- All targets share one `gate_runs` row and one log file, sectioned by
  `==== test_target: <dir> ====` banners. The gate stops at the first red
  target, and the failure summary names the crate.
- `gate_timeout_secs` bounds the whole gate, not each target.
- A target whose directory is absent from the worktree is skipped with a
  warning, so sando can still build older shas from a config describing the
  tip. If *no* target exists, the gate fails rather than reporting a pass over
  zero suites.

### What `code_smoke` builds first

Before it creates a database or boots anything, `code_smoke` compiles every
configured frontend:

```toml
[[frontend_build]]
dir = "server/frontend"

[[frontend_build]]
dir = "multithreaded/frontend"   # script = "build" by default
```

These are npm projects whose compiled output the binary serves but whose failure
`cargo build` will not report. Both MNW crates compile TypeScript from a build
script that downgrades a `tsc` error to a `cargo::warning` and lets the Rust
build succeed against whatever `static/dist/` already holds, on purpose, so a
type error in a chat widget cannot stop the forum from compiling. The cost is
that nothing downstream noticed either, and the deploy rsynced the previous
build's bundle. This is the one place that failure is fatal.

Semantics match `cargo_test`: `npm ci` first if `node_modules` is absent (usually
it is not, because the build script that produced the artifact already installed
it), stop at the first red project with the directory named, one deadline across
the gate, and a project absent from the worktree is skipped with a log line so
older shas still rebuild. Omitting the key entirely gates nothing, which is the
right default for a project with no frontend.

### The lint and supply-chain gates

`clippy` and `fmt` run over the same `[[test_target]]` list as `cargo_test`,
with the same semantics: per-target log banners, stop at the first red target
with the crate named, one deadline across the whole gate.

`cargo_audit` and `cargo_deny` are **config-gated**: a target only qualifies
once it carries a `.cargo/audit.toml` or `deny.toml`. Both tools are only
meaningful against a triaged posture, and four crates in this repo fail
`cargo audit` purely for lack of a file recording which transitive advisories
have been reviewed and accepted. Running them everywhere would make the gate
permanently and uninformatively red, which teaches everyone to ignore it.
Dropping the config file into a crate is what opts it in.

Before these existed, `-D warnings` was enforced in exactly one place
(`server/deploy/run-ci.sh`, which died with the astra pipeline) and
`cargo fmt --check` nowhere at all.

### Why `hardening_test` exists

`cargo_test` builds with `--features fast-tests`, which relaxes
`AUTH_RATE_LIMIT_BURST` (5 to 20), `SANDBOX_RATE_LIMIT_MS` (30s to 10ms), and
argon2 (46 MiB/t=2 down to 8 MiB/t=1) so the signup-heavy workflow suite
finishes in reasonable time. On top of that, the rate-limiting tests are
`#[cfg_attr(feature = "fast-tests", ignore)]`d, because a bucket refilling at
100/sec never depletes under parallel test threads.

The net effect was that Sando's only code gate skipped every test of the auth
hardening it exists to protect. `hardening_test` re-runs that suite with no
features, single-threaded, against production constants. It costs a second
compile of the server's test binary, since a different feature set is a
different cfg and shares no artifacts. It also fails closed if its name filter
matches zero tests, so renaming the suite cannot quietly turn the gate into a
green no-op.

## API

| Method | Path | Body | Purpose |
|--------|------|------|---------|
| GET    | `/state` | — | Tier list + current/previous version + last gate outcomes, plus `build` (latest build run: phase/result/failure_summary/elapsed_s, `null` until first `/rebuild`) so a poller sees in-flight/failed builds, not a frozen version |
| POST   | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). Returns `{accepted, sha, run_id}`. |
| GET    | `/runs/{id}` | — | Build-status of the run a `/rebuild` returned: `{run_id, sha, version, phase, result, failure_summary, gates[], started_at, finished_at}`. The pollable resource for a non-TUI driver — `/state` only reflects the last *successful* version. |
| GET    | `/runs/{id}/wait` | `?timeout_ms=` | Long-poll: blocks until the run settles or `timeout_ms` (default 30s, cap 120s) elapses, then returns the same `RunView`. Fire `/rebuild` → block on `/wait`. |
| POST   | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. Red post-deploy gates advance the tier (the nodes really are running it) but return 409 and flag the tier `partial` — the rollout landed, the tier cannot promote onward. |
| POST   | `/rollback/{tier}` | — | Swap `current` symlink to `previous_version` on every node in the tier. One step only: `previous_version` is cleared afterwards, so a second `/rollback` returns 409 rather than rolling forward onto the version you just escaped. |
| POST   | `/confirm/{tier}` | — | Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. |
| POST   | `/backup/fetch` | `{force?}` | Pull the prod backup. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. A fetch is rejected if the dump is under half the last verified one, which wedges the fetch permanently when the source legitimately shrinks; `force` accepts one undersized dump and makes it the new reference. `force` never skips the gzip integrity check. |
| GET    | `/events` | — | WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). |

## TUI

`sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings:

| key | action |
|-----|--------|
| ↑/↓ or j/k | select tier |
| p | `POST /promote/<selected>` (no body — version defaults to predecessor's current) |
| R | `POST /rollback/<selected>` |
| b | `POST /backup/fetch` |
| c | `POST /confirm/<selected>` |
| r | refresh hint (poller is already every 2s) |
| q / Esc / Ctrl-C | quit |

Action results show up in the events log a moment later (the actions themselves emit events from the daemon side).

## Hotfix flow

`POST /promote/{tier}` accepts:

- `hotfix: true` — skips the `burn_in` gate on the predecessor tier only. All
  other gates still apply, `manual_confirm` included: a hotfix still needs an
  operator sign-off before it reaches production.
- `reset_burn_in: true` (default `false`) — additionally nulls
  `tier_state.burn_in_started_at` on the source tier, restarting the clock
  for whatever else is still burning in there. Use this only when the hotfix
  meaningfully changes the surface area under burn-in.

## v0 limitations

- `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The
  gate drops every non-system schema on every run; do not point this at
  anything that matters.

## License

MIT. The surrounding MNW monorepo is PolyForm-Noncommercial — sando is
deliberately MIT'd because it's deploy infra, not the product.