# Sando TODO Open work only. Completed items move to `todo_done.md` (sibling file) when one exists. Design notes go in `plans/.md`, not folded into checkboxes. Format rule: every actionable line is a `- [ ]` checkbox. Headings group phases and themes; do not put status updates in them. Roadmap target: replace `server/deploy/deploy.sh` and astra-hosted `server/deploy/run-ci.sh` with Sando running on **pop-os**, gating Hetzner prod through testnot.work. **Host decision:** Sando runs on pop-os (x86_64 Ubuntu-derived, systemd). Architecturally closest to Hetzner prod, no cross-compile, no init-system split. MakeMachine and EveryCycle are now a separate project — not Sando's concern. Phases are ordered for execution. Phase 0 must finish before Phase 1 is meaningful. Phases 5+ are post-cutover hardening. ## Key Paths Read these to orient before working on Sando: - `README.md` — quickstart, API surface, v0 limitations - `sando.toml` — current topology (host → A → B; C declared, not provisioned) - `daemon/src/main.rs` — startup sequence (config → topology → migrate → sync → bare-repo bootstrap → serve) - `daemon/src/routes.rs` — `/state`, `/promote`, `/rollback`, `/rebuild`, `/backup/fetch`, `/events` - `daemon/src/gates.rs` — gate runners; the load-bearing logic - `daemon/src/build.rs` — host-tier build pipeline - `daemon/src/deploy.rs` — `deploy_local`; remote SSH stub - `daemon/migrations/001_init.sql` — schema (tiers/nodes as rows) - `server/deploy/deploy.sh` — current cross-compile + push-to-Hetzner script (what we are replacing) - `server/deploy/run-ci.sh` — current astra CI script (what we are replacing) - `_meta/docs/operations.md` — burn-in rule and hotfix policy that gates encode --- ## Phase 0 — pop-os bootstrap - [x] Provision `sando` system user on pop-os; lock down home dir; generate SSH keypair at `/srv/sando/.ssh/id_ed25519` for outbound deploys. - [x] Install scratch Postgres locally on pop-os; create `sando_scratch` role + DB used by `migration_dry_run`. (Owner of own DB; non-superuser.) - [x] Write systemd unit for `sandod` (long-run service, restart on failure, env from `/etc/sando/sando.env`). Installed at `/etc/systemd/system/sandod.service`. - [x] Write the production `sando.toml`; bare repo path under `/srv/sando/mnw.git`. Installed at `/etc/sando/sando.toml`; daemon config at `/etc/sando/sando-daemon.toml`. - [x] Install `sandod` binary at `/usr/local/bin/sandod`; enable + start the service. Live on `100.103.89.95:7766`; bare repo auto-bootstrapped at `/srv/sando/mnw.git`. - [ ] Verify MNW server builds reproducibly on pop-os. - [ ] Register sando pubkey with Hetzner prod (`deploy@alpha-west-1`) and testnot.work once that node exists. Pubkey: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEK+vhpr1V8VnsEemN9x6tAA2S05kmv/mQ3eVgSXSkJ8 sando@pop-os`. ## Phase 1 — Remote deploy The MVP only deploys to `ssh_target=local`. Production needs real SSH/rsync. - [ ] Implement `deploy::deploy_node` remote path: rsync staged binary to `:/releases//server`, then `ssh ` runs `ln -sfn releases/ current` plus `systemctl reload-or-restart `. - [ ] Add `node.service_name` to `sando.toml` (default `makenotwork.service`). - [ ] Bootstrap script for adding a fresh node: creates ``, installs the systemd unit pointing at `/current/server`, adds the sando SSH key to `authorized_keys`. Idempotent. - [ ] Garbage-collect old releases on the remote: keep last N (configurable, default 5) per node. Run at end of each successful deploy. - [ ] Handle `rsync` failure mid-deploy: leave the previous `current` symlink intact; mark `deploys.outcome = 'failed'`; do not advance `tier_state`. ## Phase 2 — Backup pipeline + migration dry-run `migration_dry_run` is the load-bearing gate. It needs a real backup source, not a fixture. - [ ] Confirm astra's offsite replica (per `sync-backup-offsite.sh`) writes a deterministic latest-link path Sando can rsync from. If not, add one. - [ ] Wire the production `sando.toml` `backup.source` to the astra rsync URL. - [ ] Schedule a daily `POST /backup/fetch` (systemd timer on pop-os) so a fresh backup is always within 24h of any promote attempt. - [ ] First end-to-end `migration_dry_run` against a real prod backup; confirm it catches the 2026-05-22 incident class (drop+recreate column migration sequence). - [ ] Document the failure modes: what does the operator see in `/state` when the dry-run fails? Capture in `plans/migration-dryrun-failures.md`. - [ ] Decide retention on `backups` table — prune rows older than N days so SQLite doesn't grow forever. ## Phase 3 — Parity with current `deploy.sh` Sando currently only ships the binary. `deploy.sh` does more. Inventory each piece and either fold it into Sando or document the explicit hand-off. - [ ] **Caddyfile** — `deploy.sh upload_config` pushes `server/deploy/Caddyfile` to `/etc/caddy/Caddyfile` and reloads Caddy. Decide: ship as a versioned config artifact alongside the binary (cleanest), or keep Caddy config out-of-band? Capture in `plans/config-artifacts.md`. - [ ] **systemd unit** — `deploy.sh` uploads `makenotwork.service`. With Sando the unit points at `current/server` and shouldn't change per release. Move unit ownership to the node-bootstrap script (Phase 1) and remove from per-deploy flow. - [ ] **Backup script** — `backup-db.sh` is uploaded by `deploy.sh`. Move to node-bootstrap; not a per-release artifact. - [ ] **Error pages** — static HTML in `server/deploy/error-pages/`. Either bake into the binary (preferred — versions with code) or ship as a `releases//error-pages/` sibling. Capture decision. - [ ] **Security configs** — `sshd-git.conf`, `fail2ban-sshd.conf`, `setup-firewall.sh`. Move to node-bootstrap. - [ ] **Restart warning** — `deploy.sh send_restart_warning` posts a banner before restart. Decide whether Sando emits this and through what surface (probably the existing in-app banner mechanism). - [ ] **Prod migrations** — today, who runs `sqlx migrate run` against prod? `deploy.sh` doesn't (verify). Sando should run prod migrations as part of `POST /promote/{tier}` for the prod tiers, OR there should be an explicit `POST /migrate/{tier}` operator action. Decide. ## Phase 4 — Cutover Run Sando in parallel with `deploy.sh` until trust is built, then retire the old path. - [ ] First successful Sando-only deploy to **testnot.work** (tier A). Old `deploy.sh` still primary for prod. - [ ] One sprint (two months) of Sando-shadow runs: every `deploy.sh` deploy is also driven through Sando in dry-run mode (gates run, deploys go to a parallel `releases/` dir on prod but don't swap `current`). Compare outcomes. - [ ] First Sando-only deploy to **Hetzner prod** (tier B). `deploy.sh` retained but unused. - [ ] Move `server/deploy/deploy.sh` to `server/deploy/archive/deploy.sh.legacy` with a header explaining the cutover; do not delete (reference for the next year). - [ ] Decommission astra CI runner (`server/deploy/run-ci.sh`). Sando's `cargo_test` gate replaces it; if any astra-specific checks are still needed (e.g., `cargo audit`), add them as additional gate kinds in `daemon/src/gates.rs`. - [ ] Update `CLAUDE.md` and `_meta/docs/operations.md` to point at Sando, not `deploy.sh`. ## Phase 5 — Operator UX The TUI polls. The MVP requires you to hand-insert a row for `manual_confirm`. Both are fine for one operator but rough. - [ ] Implement `WS /events`: tail of gate starts/finishes, deploy events, build logs. Subscribe from the TUI. - [ ] TUI: actions pane. `p` for promote (prompts for version + tier), `R` for rollback, `b` for backup fetch, `c` for manual_confirm. - [ ] `POST /confirm/{tier}` endpoint that inserts a `gate_runs` row with `passed=1, gate_kind='manual_confirm'` for the current pending version. Replaces the hand-SQL workaround. - [ ] TUI live log pane that follows the most recent build / gate run; backed by `WS /events`. - [ ] `POST /promote` body should accept `version` as optional; default to the current host version when target is A, predecessor's current when target is B+. Reduces ceremony. ## Phase 6 — Monitoring + alerting - [ ] Wire pop-os `/metrics` endpoint into the existing MNW Prometheus scrape config; record where the scrape config lives in `_meta/` or wherever monitoring already runs. - [ ] Add counters: `sando_builds_total{outcome}`, `sando_gates_total{tier,kind,outcome}`, `sando_deploys_total{tier,outcome}`, `sando_burn_in_remaining_hours{tier}`. - [ ] Alert: build failed. Page on first failure (not flap-protected — builds are infrequent). - [ ] Alert: migration_dry_run failed. Page immediately. This is the 2026-05-22-class signal. - [ ] Alert: a tier has had `current_version` unchanged for > N days while host is green. (Operator forgot to promote.) ## Phase 7 — Multi-node B+C Today B is the only prod node. Adding C is the second prod node + CF Load Balancing. - [ ] Provision tier C node (Hetzner or alternate provider — capture rationale). - [ ] Update `sando.toml`: set `c.provisioned = true`, add `[[tier.node]]`. - [ ] Set up Cloudflare Load Balancing with B + C as origin pool, health-checked. - [ ] Verify sequential canary in Sando: deploy to B, wait for CF health-check to mark healthy (probably 30-60s probe interval), then deploy to C. Add a `node.health_url` field and a gate-style wait between nodes. - [ ] Document in README that `canary = "parallel"` exists but should never be used for B+C unless you understand the failure modes. ## Phase 8 — Postgres-on-D Move Postgres off the prod app node so B+C become truly interchangeable. - [ ] Provision Postgres-only machine D (modest spec; reliability over performance). - [ ] Migrate the prod DB from Hetzner app node to D. Capture procedure in `plans/postgres-d-migration.md`. - [ ] Update `server` `DATABASE_URL` everywhere (env files on B+C, scratch URL on pop-os stays local). - [ ] Replica/HA story stays deferred; D is SPOF for now (per `_meta/preclear/.../decisions.md`). ## Phase 9 — Hardening Pick up after cutover is stable. - [ ] Tailnet ACL audit: confirm only the laptop can reach `sandod:7766`. Document the ACL. - [ ] Decide if v0.2 needs token auth on `sandod` endpoints (revisit assumption from `decisions.md` once there's a real second operator). - [ ] Sando self-deploy: Sando builds and deploys *itself* through its own pipeline. Bootstraps the bootstrap. Closes the chicken-and-egg loop and is satisfying. - [ ] Backup-of-Sando-state: nightly SQLite snapshot to astra. The state DB tracks 6 months of deploys; losing it on a pop-os disk failure would be annoying. ## Notes / non-checkbox - WS `/events` and the operator-UX work in Phase 5 can run in parallel with Phase 1-3 once Phase 0 is done. They are sequenced after for review clarity, not because they block anything. - "Hotfix override" and `reset_burn_in` flag are already implemented end-to-end (see `decisions.md`); not on this list because there's nothing left to do until prod uses them. - C tier exists in the schema as a `provisioned=false` row from day one — adding C in Phase 7 is a TOML edit, not a migration. - MakeMachine + EveryCycle are now a separate project. The hardware BOM (`plans/mm-hardware-bom.md`) should move there when that project gets its own repo.