| 1 |
# Sando TODO |
| 2 |
|
| 3 |
Open work only. Completed items move to `todo_done.md` (sibling file) when one exists. Design notes go in `plans/<name>.md`, not folded into checkboxes. |
| 4 |
|
| 5 |
Format rule: every actionable line is a `- [ ]` checkbox. Headings group phases and themes; do not put status updates in them. |
| 6 |
|
| 7 |
Roadmap target: replace `server/deploy/deploy.sh` and astra-hosted `server/deploy/run-ci.sh` with Sando running on **pop-os**, gating Hetzner prod through testnot.work. |
| 8 |
|
| 9 |
**Host decision:** Sando runs on pop-os (x86_64 Ubuntu-derived, systemd). Architecturally closest to Hetzner prod, no cross-compile, no init-system split. MakeMachine and EveryCycle are now a separate project — not Sando's concern. |
| 10 |
|
| 11 |
Phases are ordered for execution. Phase 0 must finish before Phase 1 is meaningful. Phases 5+ are post-cutover hardening. |
| 12 |
|
| 13 |
## Key Paths |
| 14 |
|
| 15 |
Read these to orient before working on Sando: |
| 16 |
|
| 17 |
- `README.md` — quickstart, API surface, v0 limitations |
| 18 |
- `sando.toml` — current topology (host → A → B; C declared, not provisioned) |
| 19 |
- `daemon/src/main.rs` — startup sequence (config → topology → migrate → sync → bare-repo bootstrap → serve) |
| 20 |
- `daemon/src/routes.rs` — `/state`, `/promote`, `/rollback`, `/rebuild`, `/backup/fetch`, `/events` |
| 21 |
- `daemon/src/gates.rs` — gate runners; the load-bearing logic |
| 22 |
- `daemon/src/build.rs` — host-tier build pipeline |
| 23 |
- `daemon/src/deploy.rs` — `deploy_local`; remote SSH stub |
| 24 |
- `daemon/migrations/001_init.sql` — schema (tiers/nodes as rows) |
| 25 |
- `server/deploy/deploy.sh` — current cross-compile + push-to-Hetzner script (what we are replacing) |
| 26 |
- `server/deploy/run-ci.sh` — current astra CI script (what we are replacing) |
| 27 |
- `_meta/docs/operations.md` — burn-in rule and hotfix policy that gates encode |
| 28 |
|
| 29 |
--- |
| 30 |
|
| 31 |
## Phase 0 — pop-os bootstrap |
| 32 |
|
| 33 |
- [x] Provision `sando` system user on pop-os; lock down home dir; generate SSH keypair at `/srv/sando/.ssh/id_ed25519` for outbound deploys. |
| 34 |
- [x] Install scratch Postgres locally on pop-os; create `sando_scratch` role + DB used by `migration_dry_run`. (Owner of own DB; non-superuser.) |
| 35 |
- [x] Write systemd unit for `sandod` (long-run service, restart on failure, env from `/etc/sando/sando.env`). Installed at `/etc/systemd/system/sandod.service`. |
| 36 |
- [x] Write the production `sando.toml`; bare repo path under `/srv/sando/mnw.git`. Installed at `/etc/sando/sando.toml`; daemon config at `/etc/sando/sando-daemon.toml`. |
| 37 |
- [x] Install `sandod` binary at `/usr/local/bin/sandod`; enable + start the service. Live on `100.103.89.95:7766`; bare repo auto-bootstrapped at `/srv/sando/mnw.git`. |
| 38 |
- [ ] Verify MNW server builds reproducibly on pop-os. |
| 39 |
- [ ] Register sando pubkey with Hetzner prod (`deploy@alpha-west-1`) and testnot.work once that node exists. Pubkey: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEK+vhpr1V8VnsEemN9x6tAA2S05kmv/mQ3eVgSXSkJ8 sando@pop-os`. |
| 40 |
|
| 41 |
## Phase 1 — Remote deploy |
| 42 |
|
| 43 |
The MVP only deploys to `ssh_target=local`. Production needs real SSH/rsync. |
| 44 |
|
| 45 |
- [ ] Implement `deploy::deploy_node` remote path: rsync staged binary to `<ssh_target>:<release_root>/releases/<version>/server`, then `ssh <ssh_target>` runs `ln -sfn releases/<version> current` plus `systemctl reload-or-restart <service>`. |
| 46 |
- [ ] Add `node.service_name` to `sando.toml` (default `makenotwork.service`). |
| 47 |
- [ ] Bootstrap script for adding a fresh node: creates `<release_root>`, installs the systemd unit pointing at `<release_root>/current/server`, adds the sando SSH key to `authorized_keys`. Idempotent. |
| 48 |
- [ ] Garbage-collect old releases on the remote: keep last N (configurable, default 5) per node. Run at end of each successful deploy. |
| 49 |
- [ ] Handle `rsync` failure mid-deploy: leave the previous `current` symlink intact; mark `deploys.outcome = 'failed'`; do not advance `tier_state`. |
| 50 |
|
| 51 |
## Phase 2 — Backup pipeline + migration dry-run |
| 52 |
|
| 53 |
`migration_dry_run` is the load-bearing gate. It needs a real backup source, not a fixture. |
| 54 |
|
| 55 |
- [ ] Confirm astra's offsite replica (per `sync-backup-offsite.sh`) writes a deterministic latest-link path Sando can rsync from. If not, add one. |
| 56 |
- [ ] Wire the production `sando.toml` `backup.source` to the astra rsync URL. |
| 57 |
- [ ] Schedule a daily `POST /backup/fetch` (systemd timer on pop-os) so a fresh backup is always within 24h of any promote attempt. |
| 58 |
- [ ] First end-to-end `migration_dry_run` against a real prod backup; confirm it catches the 2026-05-22 incident class (drop+recreate column migration sequence). |
| 59 |
- [ ] Document the failure modes: what does the operator see in `/state` when the dry-run fails? Capture in `plans/migration-dryrun-failures.md`. |
| 60 |
- [ ] Decide retention on `backups` table — prune rows older than N days so SQLite doesn't grow forever. |
| 61 |
|
| 62 |
## Phase 3 — Parity with current `deploy.sh` |
| 63 |
|
| 64 |
Sando currently only ships the binary. `deploy.sh` does more. Inventory each piece and either fold it into Sando or document the explicit hand-off. |
| 65 |
|
| 66 |
- [ ] **Caddyfile** — `deploy.sh upload_config` pushes `server/deploy/Caddyfile` to `/etc/caddy/Caddyfile` and reloads Caddy. Decide: ship as a versioned config artifact alongside the binary (cleanest), or keep Caddy config out-of-band? Capture in `plans/config-artifacts.md`. |
| 67 |
- [ ] **systemd unit** — `deploy.sh` uploads `makenotwork.service`. With Sando the unit points at `current/server` and shouldn't change per release. Move unit ownership to the node-bootstrap script (Phase 1) and remove from per-deploy flow. |
| 68 |
- [ ] **Backup script** — `backup-db.sh` is uploaded by `deploy.sh`. Move to node-bootstrap; not a per-release artifact. |
| 69 |
- [ ] **Error pages** — static HTML in `server/deploy/error-pages/`. Either bake into the binary (preferred — versions with code) or ship as a `releases/<version>/error-pages/` sibling. Capture decision. |
| 70 |
- [ ] **Security configs** — `sshd-git.conf`, `fail2ban-sshd.conf`, `setup-firewall.sh`. Move to node-bootstrap. |
| 71 |
- [ ] **Restart warning** — `deploy.sh send_restart_warning` posts a banner before restart. Decide whether Sando emits this and through what surface (probably the existing in-app banner mechanism). |
| 72 |
- [ ] **Prod migrations** — today, who runs `sqlx migrate run` against prod? `deploy.sh` doesn't (verify). Sando should run prod migrations as part of `POST /promote/{tier}` for the prod tiers, OR there should be an explicit `POST /migrate/{tier}` operator action. Decide. |
| 73 |
|
| 74 |
## Phase 4 — Cutover |
| 75 |
|
| 76 |
Run Sando in parallel with `deploy.sh` until trust is built, then retire the old path. |
| 77 |
|
| 78 |
- [ ] First successful Sando-only deploy to **testnot.work** (tier A). Old `deploy.sh` still primary for prod. |
| 79 |
- [ ] One sprint (two months) of Sando-shadow runs: every `deploy.sh` deploy is also driven through Sando in dry-run mode (gates run, deploys go to a parallel `releases/` dir on prod but don't swap `current`). Compare outcomes. |
| 80 |
- [ ] First Sando-only deploy to **Hetzner prod** (tier B). `deploy.sh` retained but unused. |
| 81 |
- [ ] Move `server/deploy/deploy.sh` to `server/deploy/archive/deploy.sh.legacy` with a header explaining the cutover; do not delete (reference for the next year). |
| 82 |
- [ ] Decommission astra CI runner (`server/deploy/run-ci.sh`). Sando's `cargo_test` gate replaces it; if any astra-specific checks are still needed (e.g., `cargo audit`), add them as additional gate kinds in `daemon/src/gates.rs`. |
| 83 |
- [ ] Update `CLAUDE.md` and `_meta/docs/operations.md` to point at Sando, not `deploy.sh`. |
| 84 |
|
| 85 |
## Phase 5 — Operator UX |
| 86 |
|
| 87 |
The TUI polls. The MVP requires you to hand-insert a row for `manual_confirm`. Both are fine for one operator but rough. |
| 88 |
|
| 89 |
- [ ] Implement `WS /events`: tail of gate starts/finishes, deploy events, build logs. Subscribe from the TUI. |
| 90 |
- [ ] TUI: actions pane. `p` for promote (prompts for version + tier), `R` for rollback, `b` for backup fetch, `c` for manual_confirm. |
| 91 |
- [ ] `POST /confirm/{tier}` endpoint that inserts a `gate_runs` row with `passed=1, gate_kind='manual_confirm'` for the current pending version. Replaces the hand-SQL workaround. |
| 92 |
- [ ] TUI live log pane that follows the most recent build / gate run; backed by `WS /events`. |
| 93 |
- [ ] `POST /promote` body should accept `version` as optional; default to the current host version when target is A, predecessor's current when target is B+. Reduces ceremony. |
| 94 |
|
| 95 |
## Phase 6 — Monitoring + alerting |
| 96 |
|
| 97 |
- [ ] Wire pop-os `/metrics` endpoint into the existing MNW Prometheus scrape config; record where the scrape config lives in `_meta/` or wherever monitoring already runs. |
| 98 |
- [ ] Add counters: `sando_builds_total{outcome}`, `sando_gates_total{tier,kind,outcome}`, `sando_deploys_total{tier,outcome}`, `sando_burn_in_remaining_hours{tier}`. |
| 99 |
- [ ] Alert: build failed. Page on first failure (not flap-protected — builds are infrequent). |
| 100 |
- [ ] Alert: migration_dry_run failed. Page immediately. This is the 2026-05-22-class signal. |
| 101 |
- [ ] Alert: a tier has had `current_version` unchanged for > N days while host is green. (Operator forgot to promote.) |
| 102 |
|
| 103 |
## Phase 7 — Multi-node B+C |
| 104 |
|
| 105 |
Today B is the only prod node. Adding C is the second prod node + CF Load Balancing. |
| 106 |
|
| 107 |
- [ ] Provision tier C node (Hetzner or alternate provider — capture rationale). |
| 108 |
- [ ] Update `sando.toml`: set `c.provisioned = true`, add `[[tier.node]]`. |
| 109 |
- [ ] Set up Cloudflare Load Balancing with B + C as origin pool, health-checked. |
| 110 |
- [ ] Verify sequential canary in Sando: deploy to B, wait for CF health-check to mark healthy (probably 30-60s probe interval), then deploy to C. Add a `node.health_url` field and a gate-style wait between nodes. |
| 111 |
- [ ] Document in README that `canary = "parallel"` exists but should never be used for B+C unless you understand the failure modes. |
| 112 |
|
| 113 |
## Phase 8 — Postgres-on-D |
| 114 |
|
| 115 |
Move Postgres off the prod app node so B+C become truly interchangeable. |
| 116 |
|
| 117 |
- [ ] Provision Postgres-only machine D (modest spec; reliability over performance). |
| 118 |
- [ ] Migrate the prod DB from Hetzner app node to D. Capture procedure in `plans/postgres-d-migration.md`. |
| 119 |
- [ ] Update `server` `DATABASE_URL` everywhere (env files on B+C, scratch URL on pop-os stays local). |
| 120 |
- [ ] Replica/HA story stays deferred; D is SPOF for now (per `_meta/preclear/.../decisions.md`). |
| 121 |
|
| 122 |
## Phase 9 — Hardening |
| 123 |
|
| 124 |
Pick up after cutover is stable. |
| 125 |
|
| 126 |
- [ ] Tailnet ACL audit: confirm only the laptop can reach `sandod:7766`. Document the ACL. |
| 127 |
- [ ] Decide if v0.2 needs token auth on `sandod` endpoints (revisit assumption from `decisions.md` once there's a real second operator). |
| 128 |
- [ ] Sando self-deploy: Sando builds and deploys *itself* through its own pipeline. Bootstraps the bootstrap. Closes the chicken-and-egg loop and is satisfying. |
| 129 |
- [ ] Backup-of-Sando-state: nightly SQLite snapshot to astra. The state DB tracks 6 months of deploys; losing it on a pop-os disk failure would be annoying. |
| 130 |
|
| 131 |
## Notes / non-checkbox |
| 132 |
|
| 133 |
- WS `/events` and the operator-UX work in Phase 5 can run in parallel with Phase 1-3 once Phase 0 is done. They are sequenced after for review clarity, not because they block anything. |
| 134 |
- "Hotfix override" and `reset_burn_in` flag are already implemented end-to-end (see `decisions.md`); not on this list because there's nothing left to do until prod uses them. |
| 135 |
- C tier exists in the schema as a `provisioned=false` row from day one — adding C in Phase 7 is a TOML edit, not a migration. |
| 136 |
- MakeMachine + EveryCycle are now a separate project. The hardware BOM (`plans/mm-hardware-bom.md`) should move there when that project gets its own repo. |
| 137 |
|