Skip to main content

max / makenotwork

sando: WIP — backup automation, events streaming, observability fixes Bundles in-flight sando work with two boot-related fixes that surfaced during the 2026-06-01 MNW launch: - gates::boot_smoke now injects DATABASE_URL (scratch DB) and SCAN_ENABLED=false. The previous spawn passed only SANDO_BOOT_SMOKE=1 and the server panicked instantly on Config::from_env's MissingDatabaseUrl, failing every rebuild. - routes::get_state falls back to the most-recently-attempted version for a tier when current_version is unset. Before, a never-green tier exposed an empty gates array via /state — debugging required SSH + direct SQLite. This is what hid the MM gate failures all morning. Broader WIP (operator's stream): backup automation (sandod-backup-fetch systemd unit + timer, sync.rs hooks), events module split, deploy.rs restart-warning + symlink swap, bootstrap-node.sh for fresh target nodes, plans docs, TUI dashboard rework, todo.md updates.
Co-Authored-By
Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-06-01 19:31 UTC
Signed with PGP, not checked
Commit: f76f08cddfd8ad1434664cbf7474c07b04fa6a2e
Parent: 869c5e0
26 files changed, +3351 insertions, -256 deletions
M sando/LICENSE +1 -1
@@ -1,6 +1,6 @@
1 1 MIT License
2 2
3 - Copyright (c) 2026 Max Jacobson
3 + Copyright (c) 2026 Make Creative, LLC
4 4
5 5 Permission is hereby granted, free of charge, to any person obtaining a copy
6 6 of this software and associated documentation files (the "Software"), to deal
M sando/README.md +22 -10
@@ -103,12 +103,29 @@
103 103 | Method | Path | Body | Purpose |
104 104 |--------|------|------|---------|
105 105 | GET | `/state` | — | Tier list + current/previous version + last gate outcomes |
106 - | POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch |
107 - | POST | `/promote/{tier}` | `{version, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state |
106 + | POST | `/rebuild` | `{sha?: string}` | Force a build; if `sha` is absent, resolves the configured deploy branch. Aborts any in-flight build (latest wins). |
107 + | POST | `/promote/{tier}` | `{version?, hotfix?, reset_burn_in?}` | Verify predecessor gates, deploy to tier nodes, advance state. `version` defaults to the predecessor tier's `current_version`. |
108 108 | POST | `/rollback/{tier}` | — | Swap `current` symlink to `previous_version` on every node in the tier |
109 - | POST | `/backup/fetch` | — | Pull the prod backup to `backup.local_path` (file:// or rsync://) |
109 + | POST | `/confirm/{tier}` | — | Insert a passing `manual_confirm` gate row for the tier's `current_version`. Replaces hand-SQL. |
110 + | POST | `/backup/fetch` | — | Pull the prod backup. Supports `file://`, `rsync://`, `ssh://user@host[:port]/path`. |
110 111 | GET | `/metrics` | — | Prometheus exposition |
111 - | GET | `/events` | — | WebSocket stream of deploy + gate events (not yet implemented) |
112 + | GET | `/events` | — | WebSocket stream of typed events (RebuildRequested, BuildStart/Ok/Failed, GateStart/Done, DeployStart/Ok/Failed, PromoteComplete, Rollback, BackupFetched, ManualConfirm, BuildAborted). |
113 +
114 + ## TUI
115 +
116 + `sando` (the TUI binary) connects to `$SANDO_DAEMON` (default `http://127.0.0.1:7766`), polls `/state` every 2s, and subscribes to `/events` over WS. Keybindings:
117 +
118 + | key | action |
119 + |-----|--------|
120 + | ↑/↓ or j/k | select tier |
121 + | p | `POST /promote/<selected>` (no body — version defaults to predecessor's current) |
122 + | R | `POST /rollback/<selected>` |
123 + | b | `POST /backup/fetch` |
124 + | c | `POST /confirm/<selected>` |
125 + | r | refresh hint (poller is already every 2s) |
126 + | q / Esc / Ctrl-C | quit |
127 +
128 + Action results show up in the events log a moment later (the actions themselves emit events from the daemon side).
112 129
113 130 ## Hotfix flow
114 131
@@ -123,14 +140,9 @@
123 140
124 141 ## v0 limitations
125 142
126 - - Remote deploys (real SSH/rsync) are stubbed. Use `ssh_target = "local"` and
127 - a local `release_root` for dev. Production wiring is a follow-up.
128 143 - `migration_dry_run` requires a scratch Postgres at `scratch_db_url`. The
129 - gate drops and recreates `public` on every run; do not point this at
144 + gate drops every non-system schema on every run; do not point this at
130 145 anything that matters.
131 - - `/events` WebSocket is not implemented; the TUI polls `/state` every 2s.
132 - - `manual_confirm` has no operator-facing trigger yet (you have to insert a
133 - `gate_runs` row with `passed=1` by hand to satisfy it).
134 146
135 147 ## License
136 148
M sando/sando.toml +6 -4
@@ -17,9 +17,10 @@
17 17 branch = "main"
18 18
19 19 [backup]
20 - # Source of the prod-backup clone used by migration_dry_run on MM.
21 - # For localhost dev this can be a file:// path to a fixture dump.
22 - source = "rsync://astra/var/backups/mnw/latest.sql.gz"
20 + # Source of the prod-backup clone used by migration_dry_run on the Sando host.
21 + # For localhost dev this can be a file:// path to a fixture dump. In prod we
22 + # pull directly from alpha-west-1 via a scoped `backup-puller` rrsync user.
23 + source = "ssh://backup-puller@alpha-west-1:2200/latest.sql.gz"
23 24 local_path = "/srv/sando/backups/latest.sql.gz"
24 25
25 26 # ---- host: pop-os local pre-staging gate ----
@@ -45,8 +46,9 @@
45 46 ]
46 47 [[tier.node]]
47 48 name = "testnot-1"
48 - ssh_target = "deploy@testnot.work"
49 + ssh_target = "deploy@testnot" # tailnet name; never the public IP / testnot.work hostname
49 50 release_root = "/opt/mnw"
51 + service_name = "makenotwork.service"
50 52
51 53 # ---- B: prod-1 ----
52 54 [[tier]]
M sando/todo.md +125 -26
@@ -4,6 +4,79 @@
4 4
5 5 Format rule: every actionable line is a `- [ ]` checkbox. Headings group phases and themes; do not put status updates in them.
6 6
7 + ## Resume here (next session)
8 +
9 + User-blocking before anything else:
10 +
11 + - [ ] Apply updated Tailscale ACL (`_private/infra/tailscale-acl-policy.json`) at https://login.tailscale.com/admin/acls — adds the `tag:server → tag:server as user max` SSH rule needed for offsite backup sync. Once live, Claude can finish: verify `makenotwork@alpha-west-1 → max@astra` ssh works, scp `MNW/server/deploy/sync-backup-offsite.sh` to `/opt/makenotwork/sync-backup-offsite.sh`, chmod +x, then trigger `sudo -u makenotwork /opt/makenotwork/backup-db.sh` and confirm a file lands in `max@astra:/opt/backups/mnw/`. Closes the "offsite broken" adjacent fire.
12 +
13 + Claude-only follow-ups (no user input needed; pick the next slice):
14 +
15 + - error-pages bake-into-binary via `include_dir!` (separate MNW PR) — closes Phase 3 §2 long-term
16 + - `cargo_test` gate red on MNW (Phase 0 follow-up) — diagnose, likely needs DB/env setup hook per test or `--test-threads=1`
17 + - Sandod build/test output streaming (Phase 0 follow-up) — pipe stdout to per-run log files instead of `Output` buffer; surface in WS `/events`
18 + - Phase 6 monitoring + alerting — Prometheus counters + alert rules
19 + - Phase 4 prep — first Sando-only deploy to testnot (needs Track B — see below)
20 + - Sando test suite — see "Testing" section below; sandod and TUI have zero unit/integration tests today
21 +
22 + Decision-gated (needs user input first):
23 +
24 + - Track B testnot live-app: postgres role+db (Claude), `.env` secrets (which Stripe/SMTP/S3 creds to use for staging — needs user), Caddyfile + Cloudflare Origin CA cert for testnot.work (user issues cert in CF dashboard; Claude installs)
25 + - Restart-warning hook for prod tier (Phase 5) — needs `CLI_SERVICE_TOKEN` accessible to sandod
26 +
27 +
28 +
29 + ## Testing
30 +
31 + Sando has zero automated tests today — daemon + TUI have been validated by running real scenarios end-to-end. Worth a pass before relying on it for prod cutover.
32 +
33 + ### TUI hands-on (Phase 5 acceptance — run interactively)
34 +
35 + - [ ] Launches against `SANDO_DAEMON=http://100.103.89.95:7766` without crashing; header shows daemon URL.
36 + - [ ] WS status: `ws ok` appears in the header within ~1s of launch (sandod is reachable).
37 + - [ ] WS reconnects: `sudo systemctl restart sandod` on pop-os; header flips `ws ok → ws ... → ws ok` within ~5s. Events resume.
38 + - [ ] `↑/↓` and `j/k` move the row highlight through all 4 tiers; selection persists across the 2s state refresh.
39 + - [ ] `b` triggers backup fetch: status bar shows `[ok] backup/fetch: ...`, events log gets a `backup_fetched` line a moment later.
40 + - [ ] `c` on tier `a` (which has `current_version=0.8.12`) records a manual_confirm; event appears.
41 + - [ ] `c` on tier `mm` (no current_version) returns an HTTP error; status bar shows `[err]`.
42 + - [ ] `p` on tier `a` (assuming gates pass) issues a real deploy; sequence of `deploy_start → deploy_ok → promote_complete` events appears.
43 + - [ ] `R` on tier `a` rolls back to `previous_version`; `rollback` event appears. Reverse with `p` again.
44 + - [ ] `q`, `Esc`, `Ctrl-C` all quit cleanly; terminal restores correctly (no leftover raw mode).
45 + - [ ] Events ring buffer trims to 200: trigger ≥200 events (loop /backup/fetch), confirm the oldest scroll out, no panic.
46 + - [ ] Action while disconnected: kill sandod, hit `b`. Status shows error, TUI stays responsive.
47 +
48 + ### Sandod unit + integration tests (Claude-only)
49 +
50 + 55 tests passing as of 2026-05-31 (14 TUI + 41 daemon). Remaining gaps:
51 +
52 + - [x] `gates::reset_scratch` — verifies dropping every non-system schema (planted `foo` + `tower_sessions`, ran reset, asserted only `public` remains). Gated by `SANDO_TEST_PG_URL` env var so it skips on hosts without postgres. Run on pop-os with `SANDO_TEST_PG_URL=postgres:///sando_scratch?host=/var/run/postgresql cargo test`.
53 + - [x] `deploy::deploy_local` — copies multiple binaries (`PRIMARY`/`ADMIN`), swaps symlink atomically across two consecutive deploys, gc_local_releases keeps last N by mtime + handles missing dir + noop under threshold. `sh_quote` round-trip.
54 + - [x] `deploy::deploy_remote` failure path — against unroutable `192.0.2.1`, verifies clean ssh-attributed error (no panic / hang); ConnectTimeout bounds the test wallclock to ~10s. Plus `deploy_node` with `ssh_target="local"` short-circuits to symlink swap.
55 + - [x] `backup::fetch` URL parsing — extracted `parse_source` → `BackupSource` enum. 10 tests: file://, rsync://, ssh:// with/without port, multi-segment ssh path, non-numeric `:foo` colon treated as part of host (not port), and all malformed-input rejections (empty, scheme-only, ftp, no path on ssh, empty user@host).
56 + - [x] `events::emit` no-subscribers no-op; `emit_reaches_a_subscriber`; envelope serializes with flat `kind` field (locks the WS/TUI contract); `lagged_subscriber_observes_recv_error_lagged` exercises broadcast capacity.
57 + - [ ] `events_ws` handler end-to-end — drive WS through a slow client, assert `{"kind":"lagged",...}` frame arrives. Possible (bind axum to ephemeral port + tungstenite client) but the bus-level lag detection is already locked in by `lagged_subscriber_observes_recv_error_lagged`. Diminishing returns vs effort. Deferred.
58 + - [ ] `build` mutex behavior — requires real cargo or a slow stub. Treated as a manual checklist item under "TUI hands-on" instead. (Already validated by hand 2026-05-31.)
59 + - [x] `routes::confirm` — rejects when tier has no `current_version` (409 Conflict — surfaced that GateBlocked maps to 409 not 400, locked in), accepts + inserts a passing gate_runs row when set, 404 on unknown tier.
60 + - [x] `routes::promote` — refuses promote-to-first-tier (409), errors when neither body nor predecessor has a version, 404 when explicit version's `versions` row is missing.
61 + - [x] `unsatisfied_gates` — 6 tests: empty, failed-kind flagging, latest-row-wins (red→green flap clears), hotfix skips burn_in only, ignores other tiers/versions, **null `passed` treated as failing** (locks the in-flight-race safety property).
62 + - [x] `run_migrator` errors on missing migrations dir.
63 + - [x] sqlx migrations exercised via existing `sync` tests.
64 +
65 + ### End-to-end harness
66 +
67 + - [ ] Single-binary smoke: spin up sandod against tmpdir config + a tmp postgres; push a fixture commit; assert the full pipeline (build → gates → MM tier_state advance) completes in under 30s. Run on CI for every sando PR.
68 + - [ ] Pre-cutover dry run: stand up a throwaway tier-B node, point production-shape config at it, run `cargo_test → migration_dry_run → boot_smoke → promote` end to end. Use existing testnot for this once Track B is done.
69 +
70 + ### TUI unit tests
71 +
72 + - [x] `format_event` — golden tests for build_ok, gate_done (pass+fail), backup_fetched, deploy_failed, unknown kind, malformed JSON.
73 + - [x] `ws_url_from`: `http://` → `ws://`, `https://` → `wss://`, only replaces scheme once, unknown scheme passes through.
74 + - [x] `Action::Display` impl produces `backup/fetch`, `promote/<tier>`, etc.
75 + - [x] `Shared::push_event` ring-buffer cap at 200; oldest entries drop in FIFO order.
76 + - [x] `truncate` short-string passthrough vs long-string ellipsis.
77 +
78 + ---
79 +
7 80 Roadmap target: replace `server/deploy/deploy.sh` and astra-hosted `server/deploy/run-ci.sh` with Sando running on **pop-os**, gating Hetzner prod through testnot.work.
8 81
9 82 **Host decision:** Sando runs on pop-os (x86_64 Ubuntu-derived, systemd). Architecturally closest to Hetzner prod, no cross-compile, no init-system split. MakeMachine and EveryCycle are now a separate project — not Sando's concern.
@@ -35,41 +108,66 @@
35 108 - [x] Write systemd unit for `sandod` (long-run service, restart on failure, env from `/etc/sando/sando.env`). Installed at `/etc/systemd/system/sandod.service`.
36 109 - [x] Write the production `sando.toml`; bare repo path under `/srv/sando/mnw.git`. Installed at `/etc/sando/sando.toml`; daemon config at `/etc/sando/sando-daemon.toml`.
37 110 - [x] Install `sandod` binary at `/usr/local/bin/sandod`; enable + start the service. Live on `100.103.89.95:7766`; bare repo auto-bootstrapped at `/srv/sando/mnw.git`.
38 - - [ ] Verify MNW server builds reproducibly on pop-os.
39 - - [ ] Register sando pubkey with Hetzner prod (`deploy@alpha-west-1`) and testnot.work once that node exists. Pubkey: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEK+vhpr1V8VnsEemN9x6tAA2S05kmv/mQ3eVgSXSkJ8 sando@pop-os`.
111 + - [x] Verify MNW server builds reproducibly on pop-os. `makenotwork` 0.8.12 built in 132s; sqlx online mode against `sando_scratch` postgres (sandod prep-resets all non-system schemas + applies all 133 MNW migrations before invoking cargo).
112 + - [ ] Register sando pubkey with Hetzner prod (`deploy@alpha-west-1`) and testnot.work once that node exists. Pubkey: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEK+vhpr1V8VnsEemN9x6tAA2S05kmv/mQ3eVgSXSkJ8 sando@pop-os`. (Moved to Phase 1 — not blocking Phase 0 exit.)
113 +
114 + ### Phase 0 follow-ups (not blocking, but visible)
115 +
116 + - [ ] `cargo_test` gate fails on MNW today — beyond the sqlx-online fix (already in), tests likely need a separate prepared DB (or per-test isolation). Investigate when wiring up Phase 1 gates.
117 + - [ ] Sandod observability: add `WS /events` (Phase 5) and consider streaming build/test stdout to a per-run log file rather than buffering in `Output`.
118 + - [ ] sqlx-cli (`v0.9.0`) at `/srv/sando/.cargo/bin/sqlx` is installed for the sando user but unused — sandod uses `sqlx::migrate::Migrator` programmatically (v0.8.6). Decide later whether to drop sqlx-cli or use it for diagnostics.
119 + - [ ] pop-os WoL: `ethtool` shows no wake-on capability on the USB ethernet — WoL likely won't work; rely on manual wake or BIOS settings. Record in `_meta/` if a solution surfaces.
40 120
41 121 ## Phase 1 — Remote deploy
42 122
43 123 The MVP only deploys to `ssh_target=local`. Production needs real SSH/rsync.
44 124
45 - - [ ] Implement `deploy::deploy_node` remote path: rsync staged binary to `<ssh_target>:<release_root>/releases/<version>/server`, then `ssh <ssh_target>` runs `ln -sfn releases/<version> current` plus `systemctl reload-or-restart <service>`.
46 - - [ ] Add `node.service_name` to `sando.toml` (default `makenotwork.service`).
47 - - [ ] Bootstrap script for adding a fresh node: creates `<release_root>`, installs the systemd unit pointing at `<release_root>/current/server`, adds the sando SSH key to `authorized_keys`. Idempotent.
48 - - [ ] Garbage-collect old releases on the remote: keep last N (configurable, default 5) per node. Run at end of each successful deploy.
49 - - [ ] Handle `rsync` failure mid-deploy: leave the previous `current` symlink intact; mark `deploys.outcome = 'failed'`; do not advance `tier_state`.
125 + - [x] Implement `deploy::deploy_node` remote path: rsync staged binary to `<ssh_target>:<release_root>/releases/<version>/<bin_name>`, then `ssh <ssh_target>` does `mv -Tf` symlink swap + `sudo systemctl reload-or-restart <service>`. First real promote landed 2026-05-31: pop-os → testnot, version 0.8.12.
126 + - [x] Add `node.service_name` to `sando.toml` (default `makenotwork.service`).
127 + - [x] Bootstrap script for adding a fresh node: `MNW/sando/deploy/bootstrap-node.sh`. (See Phase 3 — node-bootstrap script for full details.)
128 + - [x] Garbage-collect old releases on the remote: keep last N=5 per node, sorted by mtime. Runs at end of each successful deploy (local + remote variants). Tied via `RELEASES_TO_KEEP` const.
129 + - [x] Handle `rsync` failure mid-deploy: leave the previous `current` symlink intact; mark `deploys.outcome = 'failed'`; do not advance `tier_state`. (Verified the routes.rs path; rsync runs before symlink swap so failure naturally leaves `current` untouched.)
130 +
131 + ### Phase 1 — Track B: testnot live-app setup (NOT blocking Phase 2)
132 +
133 + Sando's deploy machinery is done, but testnot's MNW runtime needs the rest before its `makenotwork.service` can stay up:
134 +
135 + - [ ] Provision `makenotwork` postgres role + db on testnot (postgres-18 already installed).
136 + - [ ] `/opt/mnw/.env` with staging Stripe keys, SMTP, S3, DATABASE_URL, all other MNW env. Decide which subset of integrations get test/sandbox credentials vs are stubbed.
137 + - [ ] Caddyfile for testnot.work — strip prod's blocks down to just the main reverse_proxy (and forums/cdn if needed). Cloudflare Origin CA cert for testnot.work issued + placed at `/etc/caddy/`. AOP CA already universal.
138 + - [ ] `error-pages/` for testnot (copy or symlink from a release dir).
139 + - [ ] Wire post-deploy smoke check (`curl https://testnot.work/health` after the symlink swap, before declaring deploy ok). Sando-side, gate-like; spec in Phase 2 boot_smoke wording.
50 140
51 141 ## Phase 2 — Backup pipeline + migration dry-run
52 142
53 143 `migration_dry_run` is the load-bearing gate. It needs a real backup source, not a fixture.
54 144
55 - - [ ] Confirm astra's offsite replica (per `sync-backup-offsite.sh`) writes a deterministic latest-link path Sando can rsync from. If not, add one.
56 - - [ ] Wire the production `sando.toml` `backup.source` to the astra rsync URL.
57 - - [ ] Schedule a daily `POST /backup/fetch` (systemd timer on pop-os) so a fresh backup is always within 24h of any promote attempt.
58 - - [ ] First end-to-end `migration_dry_run` against a real prod backup; confirm it catches the 2026-05-22 incident class (drop+recreate column migration sequence).
59 - - [ ] Document the failure modes: what does the operator see in `/state` when the dry-run fails? Capture in `plans/migration-dryrun-failures.md`.
60 - - [ ] Decide retention on `backups` table — prune rows older than N days so SQLite doesn't grow forever.
145 + - [x] ~~Confirm astra's offsite replica writes a deterministic latest-link path.~~ Pivoted: pull direct from prod (`backup-puller@alpha-west-1:2200`, rrsync-locked to `/opt/makenotwork/backups/`). Astra offsite is separately broken — see carryover below.
146 + - [x] Wire the production `sando.toml` `backup.source` — `ssh://backup-puller@alpha-west-1:2200/latest.sql.gz` with `latest.sql.gz` as a hard link on prod.
147 + - [x] Schedule a daily `POST /backup/fetch` (systemd timer on pop-os). `sandod-backup-fetch.{service,timer}` in `MNW/sando/deploy/`. Runs daily at 04:00 UTC (one hour after prod's 03:00 UTC backup-db.sh). Service uses `EnvironmentFile=/etc/sando/sando.env` for `$SANDO_DAEMON`. Verified 2026-05-31: one-shot test pulled 36MB backup, recorded in `backups` table.
148 + - [x] First end-to-end `migration_dry_run` against a real prod backup. Passed 2026-05-31 for sha 4541ebc in 1.2s: restored 36MB dump + applied all 133 migrations cleanly. Sha eee96a7 correctly failed `migration_dry_run` because it lacked migrations 123-132 that prod has applied — exactly the prod-vs-repo drift the gate is designed to catch.
149 + - [x] Document the failure modes: `plans/migration-dryrun-failures.md`. Covers all 7 fail modes (no backup, scratch_url unset, scratch reset, restore, drift, checksum mismatch, content broken against prod data) with operator playbook.
150 + - [x] Decide retention on `backups` table. 30 days; pruned at end of `backup::fetch`. `DELETE FROM backups WHERE fetched_at < datetime('now', '-30 days')`.
151 +
152 + ### Phase 2 carryovers / adjacent fires
153 +
154 + - [ ] **Offsite backup sync from prod → astra still broken.** Diagnosed 2026-05-31: `sync-backup-offsite.sh` was never deployed to prod (`deploy.sh` gap when it was added). `makenotwork@prod` had no SSH key. Generated key + installed pubkey on `max@astra:~/.ssh/authorized_keys`, created `/opt/backups/mnw` on astra. **Blocked** on Tailscale ACL: astra runs only Tailscale SSH (no regular sshd on a bypass port), and the ACL denies `tag:tagged-devices` (alpha-west-1) → astra as user `max`. Needs ACL update in the Tailscale admin console, then deploy `sync-backup-offsite.sh` to `/opt/makenotwork/` and test. Makenotwork@prod pubkey: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzyQQ7pmBIZat8fABlpG/opwh4w5GhLIfkX2qxKxuT0 makenotwork@alpha-west-1`.
155 + - [x] **Prod backup `latest.sql.gz` hard link.** `backup-db.sh` now maintains `latest.sql.gz` atomically (`ln -f $LATEST.new && mv -Tf .new latest.sql.gz`). Deployed 2026-05-31; manual run verified (nlinks=2).
61 156
62 157 ## Phase 3 — Parity with current `deploy.sh`
63 158
64 - Sando currently only ships the binary. `deploy.sh` does more. Inventory each piece and either fold it into Sando or document the explicit hand-off.
159 + Decisions captured in `plans/config-artifacts.md`. Summary: Caddyfile / systemd unit / backup script / security configs all move to **one-time node-bootstrap**, not per-deploy. error-pages bake into binary (MNW PR) with sibling fallback. mnw-admin ships alongside server via `bin_names: Vec<String>`. Restart warning is Phase 5, prod-tier-only. Prod migrations: server self-applies on startup (`main.rs:73`), sando does not.
65 160
66 - - [ ] **Caddyfile** — `deploy.sh upload_config` pushes `server/deploy/Caddyfile` to `/etc/caddy/Caddyfile` and reloads Caddy. Decide: ship as a versioned config artifact alongside the binary (cleanest), or keep Caddy config out-of-band? Capture in `plans/config-artifacts.md`.
67 - - [ ] **systemd unit** — `deploy.sh` uploads `makenotwork.service`. With Sando the unit points at `current/server` and shouldn't change per release. Move unit ownership to the node-bootstrap script (Phase 1) and remove from per-deploy flow.
68 - - [ ] **Backup script** — `backup-db.sh` is uploaded by `deploy.sh`. Move to node-bootstrap; not a per-release artifact.
69 - - [ ] **Error pages** — static HTML in `server/deploy/error-pages/`. Either bake into the binary (preferred — versions with code) or ship as a `releases/<version>/error-pages/` sibling. Capture decision.
70 - - [ ] **Security configs** — `sshd-git.conf`, `fail2ban-sshd.conf`, `setup-firewall.sh`. Move to node-bootstrap.
71 - - [ ] **Restart warning** — `deploy.sh send_restart_warning` posts a banner before restart. Decide whether Sando emits this and through what surface (probably the existing in-app banner mechanism).
72 - - [ ] **Prod migrations** — today, who runs `sqlx migrate run` against prod? `deploy.sh` doesn't (verify). Sando should run prod migrations as part of `POST /promote/{tier}` for the prod tiers, OR there should be an explicit `POST /migrate/{tier}` operator action. Decide.
161 + - [x] **Caddyfile** — decided: bootstrap-only. Not per-deploy. (`plans/config-artifacts.md` §1.)
162 + - [x] **systemd unit** — decided: bootstrap-only. (§4.)
163 + - [x] **Backup script** — decided: bootstrap-only. (§6.)
164 + - [x] **Error pages** — short-term done: ship as release-dir sibling. `build_and_run_mm` `cp -a` from `worktree/server/deploy/error-pages/` into the staged release dir; deploy_node's rsync of the whole dir picks it up. Verified on testnot 2026-05-31. Long-term `include_dir!` bake-in still a separate MNW PR.
165 + - [x] **mnw-admin binary** — `cfg.bin_names: Vec<String>` (default `["server"]`, MNW uses `["makenotwork","mnw-admin"]`). `deploy_local` copies each from worktree's `target/release/<bin>`; `deploy_node` rsyncs the whole staged dir. `Config::primary_bin()` returns first entry for systemd reference. `versions.artifact_path` stores the primary; release dir is derived as `.parent()`. Verified on testnot 2026-05-31.
166 + - [x] **Security configs** — decided: bootstrap-only. (§5.)
167 + - [ ] **Restart warning** — Phase 5, prod-tier only via `tier.restart_warning_seconds` in `sando.toml`; needs `CLI_SERVICE_TOKEN` in `/etc/sando/sando.env`. (§7.)
168 + - [x] **Cross-compile from macOS** — decided: retire after one sprint of testnot parity verification. Pop-os builds natively. (§8.)
169 + - [x] **Prod migrations** — decided: server self-applies on startup. Sando does NOT run them. `migration_dry_run` gate is the prod safety net. (§9.)
170 + - [x] **Node-bootstrap script** — `MNW/sando/deploy/bootstrap-node.sh`. Idempotent. Takes `SANDO_PUBKEY` (required), `BIN_NAME`, `SERVICE_NAME`, `SERVICE_USER`, `DEPLOY_ROOT` env. Installs base packages (rsync/ufw/fail2ban), optionally postgres/tailscale/caddy, creates deploy user + dirs + sudoers entry + systemd unit, sets up UFW. Deliberately does NOT touch Caddyfile content, certs, postgres role/db, or secrets — those are operator-decisions per-node. testnot was done by hand and matches roughly what the script produces. Test by re-running on the next node added (tier B Hetzner prod move or tier C).
73 171
74 172 ## Phase 4 — Cutover
75 173
@@ -86,11 +184,12 @@
86 184
87 185 The TUI polls. The MVP requires you to hand-insert a row for `manual_confirm`. Both are fine for one operator but rough.
88 186
89 - - [ ] Implement `WS /events`: tail of gate starts/finishes, deploy events, build logs. Subscribe from the TUI.
90 - - [ ] TUI: actions pane. `p` for promote (prompts for version + tier), `R` for rollback, `b` for backup fetch, `c` for manual_confirm.
91 - - [ ] `POST /confirm/{tier}` endpoint that inserts a `gate_runs` row with `passed=1, gate_kind='manual_confirm'` for the current pending version. Replaces the hand-SQL workaround.
92 - - [ ] TUI live log pane that follows the most recent build / gate run; backed by `WS /events`.
93 - - [ ] `POST /promote` body should accept `version` as optional; default to the current host version when target is A, predecessor's current when target is B+. Reduces ceremony.
187 + - [x] Build mutex: single-slot `AppState.active_build: Mutex<Option<AbortHandle>>`; newer `/rebuild` aborts any in-flight build. Cargo commands set `.kill_on_drop(true)` so abort propagates SIGKILL to cargo + rustc children. (Landed 2026-05-31 after observing two concurrent builds racing the scratch DB.)
188 + - [x] Implement `WS /events`: tail of gate starts/finishes, deploy events, build logs. Event enum in `daemon/src/events.rs`; `broadcast::channel(256)` in `AppState`; emit sites in build.rs, gates.rs, routes.rs (rebuild, promote, rollback, confirm, backup_fetch). Verified 2026-05-31: live JSON envelopes stream to a python `websockets` client.
189 + - [x] TUI: actions pane. `↑↓`/`jk` select tier; `p` promote (no body — defaults version); `R` rollback; `b` backup fetch; `c` manual_confirm. Action results land in the events log. Daemon URL via `$SANDO_DAEMON`. Built in `tui/src/main.rs` 2026-05-31.
190 + - [x] `POST /confirm/{tier}` endpoint — inserts `gate_runs` row with `passed=1, gate_kind='manual_confirm'` for the tier's `current_version`. Replaces hand-SQL workaround. Verified 2026-05-31 against tier `a`.
191 + - [x] TUI live log pane that follows the most recent build / gate run; backed by `WS /events`. 200-event ring buffer, human-formatted per kind. WS auto-reconnects every 3s. Header shows ws connection state.
192 + - [x] `POST /promote` body — `version` now optional; defaults to predecessor tier's `current_version`. (Unblocks the "promote what just baked" flow.)
94 193
95 194 ## Phase 6 — Monitoring + alerting
96 195
M sando/daemon/Cargo.lock +337 -2
@@ -184,6 +184,12 @@
184 184 source = "registry+https://github.com/rust-lang/crates.io-index"
185 185 checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
186 186
187 + [[package]]
188 + name = "cfg_aliases"
189 + version = "0.2.1"
190 + source = "registry+https://github.com/rust-lang/crates.io-index"
191 + checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
192 +
187 193 [[package]]
188 194 name = "chrono"
189 195 version = "0.4.44"
@@ -381,6 +387,12 @@
381 387 "smallvec",
382 388 ]
383 389
390 + [[package]]
391 + name = "fastrand"
392 + version = "2.4.1"
393 + source = "registry+https://github.com/rust-lang/crates.io-index"
394 + checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
395 +
384 396 [[package]]
385 397 name = "find-msvc-tools"
386 398 version = "0.1.9"
@@ -522,8 +534,10 @@
522 534 checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
523 535 dependencies = [
524 536 "cfg-if",
537 + "js-sys",
525 538 "libc",
526 539 "wasi",
540 + "wasm-bindgen",
527 541 ]
528 542
529 543 [[package]]
@@ -533,9 +547,11 @@
533 547 checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
534 548 dependencies = [
535 549 "cfg-if",
550 + "js-sys",
536 551 "libc",
537 552 "r-efi",
538 553 "wasip2",
554 + "wasm-bindgen",
539 555 ]
540 556
541 557 [[package]]
@@ -681,6 +697,23 @@
681 697 "pin-project-lite",
682 698 "smallvec",
683 699 "tokio",
700 + "want",
701 + ]
702 +
703 + [[package]]
704 + name = "hyper-rustls"
705 + version = "0.27.9"
706 + source = "registry+https://github.com/rust-lang/crates.io-index"
707 + checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
708 + dependencies = [
709 + "http",
710 + "hyper",
711 + "hyper-util",
712 + "rustls",
713 + "tokio",
714 + "tokio-rustls",
715 + "tower-service",
716 + "webpki-roots",
684 717 ]
685 718
686 719 [[package]]
@@ -689,13 +722,21 @@
689 722 source = "registry+https://github.com/rust-lang/crates.io-index"
690 723 checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
691 724 dependencies = [
725 + "base64",
692 726 "bytes",
727 + "futures-channel",
728 + "futures-util",
693 729 "http",
694 730 "http-body",
695 731 "hyper",
732 + "ipnet",
733 + "libc",
734 + "percent-encoding",
696 735 "pin-project-lite",
736 + "socket2",
697 737 "tokio",
698 738 "tower-service",
739 + "tracing",
699 740 ]
700 741
701 742 [[package]]
@@ -835,6 +876,12 @@
835 876 "hashbrown 0.17.1",
836 877 ]
837 878
879 + [[package]]
880 + name = "ipnet"
881 + version = "2.12.0"
882 + source = "registry+https://github.com/rust-lang/crates.io-index"
883 + checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
884 +
838 885 [[package]]
839 886 name = "itoa"
840 887 version = "1.0.18"
@@ -908,6 +955,12 @@
908 955 "vcpkg",
909 956 ]
910 957
958 + [[package]]
959 + name = "linux-raw-sys"
960 + version = "0.12.1"
961 + source = "registry+https://github.com/rust-lang/crates.io-index"
962 + checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
963 +
911 964 [[package]]
912 965 name = "litemap"
913 966 version = "0.8.2"
@@ -942,6 +995,12 @@
942 995 "tracing-subscriber",
943 996 ]
944 997
998 + [[package]]
999 + name = "lru-slab"
1000 + version = "0.1.2"
1001 + source = "registry+https://github.com/rust-lang/crates.io-index"
1002 + checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
1003 +
945 1004 [[package]]
946 1005 name = "matchers"
947 1006 version = "0.2.0"
@@ -1224,6 +1283,61 @@
1224 1283 "winapi",
1225 1284 ]
1226 1285
1286 + [[package]]
1287 + name = "quinn"
1288 + version = "0.11.9"
1289 + source = "registry+https://github.com/rust-lang/crates.io-index"
1290 + checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
1291 + dependencies = [
1292 + "bytes",
1293 + "cfg_aliases",
1294 + "pin-project-lite",
1295 + "quinn-proto",
1296 + "quinn-udp",
1297 + "rustc-hash",
1298 + "rustls",
1299 + "socket2",
1300 + "thiserror",
1301 + "tokio",
1302 + "tracing",
1303 + "web-time",
1304 + ]
1305 +
1306 + [[package]]
1307 + name = "quinn-proto"
1308 + version = "0.11.14"
1309 + source = "registry+https://github.com/rust-lang/crates.io-index"
1310 + checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
1311 + dependencies = [
1312 + "bytes",
1313 + "getrandom 0.3.4",
1314 + "lru-slab",
1315 + "rand 0.9.4",
1316 + "ring",
1317 + "rustc-hash",
1318 + "rustls",
1319 + "rustls-pki-types",
1320 + "slab",
1321 + "thiserror",
1322 + "tinyvec",
1323 + "tracing",
1324 + "web-time",
1325 + ]
1326 +
1327 + [[package]]
1328 + name = "quinn-udp"
1329 + version = "0.5.14"
1330 + source = "registry+https://github.com/rust-lang/crates.io-index"
1331 + checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
1332 + dependencies = [
1333 + "cfg_aliases",
1334 + "libc",
1335 + "once_cell",
1336 + "socket2",
1337 + "tracing",
1338 + "windows-sys 0.60.2",
1339 + ]
1340 +
1227 1341 [[package]]
1228 1342 name = "quote"
1229 1343 version = "1.0.45"
@@ -1360,6 +1474,58 @@
1360 1474 source = "registry+https://github.com/rust-lang/crates.io-index"
1361 1475 checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
1362 1476
1477 + [[package]]
1478 + name = "reqwest"
1479 + version = "0.12.28"
1480 + source = "registry+https://github.com/rust-lang/crates.io-index"
1481 + checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
1482 + dependencies = [
1483 + "base64",
1484 + "bytes",
1485 + "futures-core",
1486 + "http",
1487 + "http-body",
1488 + "http-body-util",
1489 + "hyper",
1490 + "hyper-rustls",
1491 + "hyper-util",
1492 + "js-sys",
1493 + "log",
1494 + "percent-encoding",
1495 + "pin-project-lite",
1496 + "quinn",
1497 + "rustls",
1498 + "rustls-pki-types",
1499 + "serde",
1500 + "serde_json",
1501 + "serde_urlencoded",
1502 + "sync_wrapper",
1503 + "tokio",
1504 + "tokio-rustls",
1505 + "tower",
1506 + "tower-http",
1507 + "tower-service",
1508 + "url",
1509 + "wasm-bindgen",
1510 + "wasm-bindgen-futures",
1511 + "web-sys",
1512 + "webpki-roots",
1513 + ]
1514 +
1515 + [[package]]
1516 + name = "ring"
1517 + version = "0.17.14"
1518 + source = "registry+https://github.com/rust-lang/crates.io-index"
1519 + checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
1520 + dependencies = [
1521 + "cc",
1522 + "cfg-if",
1523 + "getrandom 0.2.17",
1524 + "libc",
1525 + "untrusted",
1526 + "windows-sys 0.52.0",
1527 + ]
1528 +
1363 1529 [[package]]
1364 1530 name = "rsa"
1365 1531 version = "0.9.10"
@@ -1380,6 +1546,60 @@
1380 1546 "zeroize",
1381 1547 ]
1382 1548
1549 + [[package]]
1550 + name = "rustc-hash"
1551 + version = "2.1.2"
1552 + source = "registry+https://github.com/rust-lang/crates.io-index"
1553 + checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
1554 +
1555 + [[package]]
1556 + name = "rustix"
1557 + version = "1.1.4"
1558 + source = "registry+https://github.com/rust-lang/crates.io-index"
1559 + checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
1560 + dependencies = [
1561 + "bitflags",
1562 + "errno",
1563 + "libc",
1564 + "linux-raw-sys",
1565 + "windows-sys 0.61.2",
1566 + ]
1567 +
1568 + [[package]]
1569 + name = "rustls"
1570 + version = "0.23.40"
1571 + source = "registry+https://github.com/rust-lang/crates.io-index"
1572 + checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
1573 + dependencies = [
1574 + "once_cell",
1575 + "ring",
1576 + "rustls-pki-types",
1577 + "rustls-webpki",
1578 + "subtle",
1579 + "zeroize",
1580 + ]
1581 +
1582 + [[package]]
1583 + name = "rustls-pki-types"
1584 + version = "1.14.1"
1585 + source = "registry+https://github.com/rust-lang/crates.io-index"
1586 + checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
1587 + dependencies = [
1588 + "web-time",
1589 + "zeroize",
1590 + ]
1591 +
1592 + [[package]]
1593 + name = "rustls-webpki"
1594 + version = "0.103.13"
1595 + source = "registry+https://github.com/rust-lang/crates.io-index"
1596 + checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
1597 + dependencies = [
1598 + "ring",
1599 + "rustls-pki-types",
1600 + "untrusted",
1601 + ]
1602 +
1383 1603 [[package]]
1384 1604 name = "rustversion"
1385 1605 version = "1.0.22"
@@ -1399,14 +1619,18 @@
1399 1619 "anyhow",
1400 1620 "axum",
1401 1621 "chrono",
1622 + "http-body-util",
1402 1623 "metrics",
1403 1624 "metrics-exporter-prometheus",
1625 + "reqwest",
1404 1626 "serde",
1405 1627 "serde_json",
1406 1628 "sqlx",
1629 + "tempfile",
1407 1630 "thiserror",
1408 1631 "tokio",
1409 1632 "toml",
1633 + "tower",
1410 1634 "tracing",
1411 1635 "tracing-subscriber",
1412 1636 ]
@@ -1836,6 +2060,9 @@
1836 2060 version = "1.0.2"
1837 2061 source = "registry+https://github.com/rust-lang/crates.io-index"
1838 2062 checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
2063 + dependencies = [
2064 + "futures-core",
2065 + ]
1839 2066
1840 2067 [[package]]
1841 2068 name = "synstructure"
@@ -1848,6 +2075,19 @@
1848 2075 "syn",
1849 2076 ]
1850 2077
2078 + [[package]]
2079 + name = "tempfile"
2080 + version = "3.27.0"
2081 + source = "registry+https://github.com/rust-lang/crates.io-index"
2082 + checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
2083 + dependencies = [
2084 + "fastrand",
2085 + "getrandom 0.3.4",
2086 + "once_cell",
2087 + "rustix",
2088 + "windows-sys 0.61.2",
2089 + ]
2090 +
1851 2091 [[package]]
1852 2092 name = "thiserror"
1853 2093 version = "2.0.18"
@@ -1929,6 +2169,16 @@
1929 2169 "syn",
1930 2170 ]
1931 2171
2172 + [[package]]
2173 + name = "tokio-rustls"
2174 + version = "0.26.4"
2175 + source = "registry+https://github.com/rust-lang/crates.io-index"
2176 + checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
2177 + dependencies = [
2178 + "rustls",
2179 + "tokio",
2180 + ]
2181 +
1932 2182 [[package]]
1933 2183 name = "tokio-stream"
1934 2184 version = "0.1.18"
@@ -2009,6 +2259,24 @@
2009 2259 "tracing",
2010 2260 ]
2011 2261
2262 + [[package]]
2263 + name = "tower-http"
2264 + version = "0.6.11"
2265 + source = "registry+https://github.com/rust-lang/crates.io-index"
2266 + checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
2267 + dependencies = [
2268 + "bitflags",
2269 + "bytes",
2270 + "futures-util",
2271 + "http",
2272 + "http-body",
2273 + "pin-project-lite",
2274 + "tower",
2275 + "tower-layer",
2276 + "tower-service",
2277 + "url",
2278 + ]
2279 +
2012 2280 [[package]]
2013 2281 name = "tower-layer"
2014 2282 version = "0.3.3"
@@ -2096,6 +2364,12 @@
2096 2364 "tracing-serde",
2097 2365 ]
2098 2366
2367 + [[package]]
2368 + name = "try-lock"
2369 + version = "0.2.5"
2370 + source = "registry+https://github.com/rust-lang/crates.io-index"
2371 + checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
2372 +
2099 2373 [[package]]
2100 2374 name = "tungstenite"
2101 2375 version = "0.29.0"
@@ -2145,6 +2419,12 @@
2145 2419 source = "registry+https://github.com/rust-lang/crates.io-index"
2146 2420 checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d"
2147 2421
2422 + [[package]]
2423 + name = "untrusted"
2424 + version = "0.9.0"
2425 + source = "registry+https://github.com/rust-lang/crates.io-index"
2426 + checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
2427 +
2148 2428 [[package]]
2149 2429 name = "url"
2150 2430 version = "2.5.8"
@@ -2181,6 +2461,15 @@
2181 2461 source = "registry+https://github.com/rust-lang/crates.io-index"
2182 2462 checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
2183 2463
2464 + [[package]]
2465 + name = "want"
2466 + version = "0.3.1"
2467 + source = "registry+https://github.com/rust-lang/crates.io-index"
2468 + checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
2469 + dependencies = [
2470 + "try-lock",
2471 + ]
2472 +
2184 2473 [[package]]
2185 2474 name = "wasi"
2186 2475 version = "0.11.1+wasi-snapshot-preview1"
@@ -2215,6 +2504,16 @@
2215 2504 "wasm-bindgen-shared",
2216 2505 ]
2217 2506
2507 + [[package]]
2508 + name = "wasm-bindgen-futures"
2509 + version = "0.4.72"
2510 + source = "registry+https://github.com/rust-lang/crates.io-index"
2511 + checksum = "9473dbd2991ae90b6291c3c32c30c6187ac49aa32f9905d1cce280ec1e110b0f"
2512 + dependencies = [
2513 + "js-sys",
2514 + "wasm-bindgen",
2515 + ]
2516 +
2218 2517 [[package]]
2219 2518 name = "wasm-bindgen-macro"
2220 2519 version = "0.2.122"
@@ -2257,6 +2556,25 @@
2257 2556 "wasm-bindgen",
2258 2557 ]
2259 2558
2559 + [[package]]
2560 + name = "web-time"
2561 + version = "1.1.0"
2562 + source = "registry+https://github.com/rust-lang/crates.io-index"
2563 + checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
2564 + dependencies = [
2565 + "js-sys",
2566 + "wasm-bindgen",
2567 + ]
2568 +
2569 + [[package]]
2570 + name = "webpki-roots"
2571 + version = "1.0.7"
2572 + source = "registry+https://github.com/rust-lang/crates.io-index"
2573 + checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d"
2574 + dependencies = [
2575 + "rustls-pki-types",
2576 + ]
2577 +
2260 2578 [[package]]
2261 2579 name = "whoami"
2262 2580 version = "1.6.1"
@@ -2354,7 +2672,25 @@
2354 2672 source = "registry+https://github.com/rust-lang/crates.io-index"
2355 2673 checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
2356 2674 dependencies = [
2357 - "windows-targets",
2675 + "windows-targets 0.48.5",
2676 + ]
2677 +
2678 + [[package]]
2679 + name = "windows-sys"
2680 + version = "0.52.0"
2681 + source = "registry+https://github.com/rust-lang/crates.io-index"
2682 + checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
2683 + dependencies = [
2684 + "windows-targets 0.52.6",
2685 + ]
2686 +
2687 + [[package]]
2688 + name = "windows-sys"
2689 + version = "0.60.2"
2690 + source = "registry+https://github.com/rust-lang/crates.io-index"
2691 + checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
2692 + dependencies = [
2693 + "windows-targets 0.53.5",
2358 2694 ]
2359 2695
2360 2696 [[package]]
@@ -2372,13 +2708,46 @@
2372 2708 source = "registry+https://github.com/rust-lang/crates.io-index"
2373 2709 checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c"
2374 2710 dependencies = [
2375 - "windows_aarch64_gnullvm",
Lines truncated
M sando/daemon/Cargo.toml +6
@@ -22,3 +22,9 @@
22 22 anyhow = "1.0.102"
23 23 thiserror = "2.0.18"
24 24 chrono = { version = "0.4", features = ["serde"] }
25 +
26 + [dev-dependencies]
27 + tempfile = "3.20"
28 + tower = { version = "0.5", features = ["util"] }
29 + http-body-util = "0.1"
30 + reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
M sando/deploy/sando-daemon.toml.example +2 -1
@@ -5,5 +5,6 @@
5 5 db_path = "/srv/sando/state/sando.db"
6 6 topology_path = "/etc/sando/sando.toml"
7 7 workdir = "/srv/sando/work"
8 - release_root = "/srv/sando/releases"
8 + release_root = "/srv/sando"
9 9 scratch_db_url = "postgres:///sando_scratch?host=/var/run/postgresql"
10 + bin_names = ["makenotwork", "mnw-admin"]
M sando/tui/Cargo.lock +304 -9
@@ -8,6 +8,15 @@
8 8 source = "registry+https://github.com/rust-lang/crates.io-index"
9 9 checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
10 10
11 + [[package]]
12 + name = "android_system_properties"
13 + version = "0.1.5"
14 + source = "registry+https://github.com/rust-lang/crates.io-index"
15 + checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
16 + dependencies = [
17 + "libc",
18 + ]
19 +
11 20 [[package]]
12 21 name = "anyhow"
13 22 version = "1.0.102"
@@ -20,6 +29,12 @@
20 29 source = "registry+https://github.com/rust-lang/crates.io-index"
21 30 checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
22 31
32 + [[package]]
33 + name = "autocfg"
34 + version = "1.5.1"
35 + source = "registry+https://github.com/rust-lang/crates.io-index"
36 + checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
37 +
23 38 [[package]]
24 39 name = "base64"
25 40 version = "0.22.1"
@@ -32,12 +47,27 @@
32 47 source = "registry+https://github.com/rust-lang/crates.io-index"
33 48 checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3"
34 49
50 + [[package]]
51 + name = "block-buffer"
52 + version = "0.10.4"
53 + source = "registry+https://github.com/rust-lang/crates.io-index"
54 + checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
55 + dependencies = [
56 + "generic-array",
57 + ]
58 +
35 59 [[package]]
36 60 name = "bumpalo"
37 61 version = "3.20.3"
38 62 source = "registry+https://github.com/rust-lang/crates.io-index"
39 63 checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
40 64
65 + [[package]]
66 + name = "byteorder"
67 + version = "1.5.0"
68 + source = "registry+https://github.com/rust-lang/crates.io-index"
69 + checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
70 +
41 71 [[package]]
42 72 name = "bytes"
43 73 version = "1.11.1"
@@ -81,6 +111,18 @@
81 111 source = "registry+https://github.com/rust-lang/crates.io-index"
82 112 checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
83 113
114 + [[package]]
115 + name = "chrono"
116 + version = "0.4.44"
117 + source = "registry+https://github.com/rust-lang/crates.io-index"
118 + checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0"
119 + dependencies = [
120 + "iana-time-zone",
121 + "num-traits",
122 + "serde",
123 + "windows-link",
124 + ]
125 +
84 126 [[package]]
85 127 name = "compact_str"
86 128 version = "0.8.1"
@@ -95,6 +137,21 @@
95 137 "static_assertions",
96 138 ]
97 139
140 + [[package]]
141 + name = "core-foundation-sys"
142 + version = "0.8.7"
143 + source = "registry+https://github.com/rust-lang/crates.io-index"
144 + checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
145 +
146 + [[package]]
147 + name = "cpufeatures"
148 + version = "0.2.17"
149 + source = "registry+https://github.com/rust-lang/crates.io-index"
150 + checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
151 + dependencies = [
152 + "libc",
153 + ]
154 +
98 155 [[package]]
99 156 name = "crossterm"
100 157 version = "0.28.1"
@@ -120,6 +177,16 @@
120 177 "winapi",
121 178 ]
122 179
180 + [[package]]
181 + name = "crypto-common"
182 + version = "0.1.7"
183 + source = "registry+https://github.com/rust-lang/crates.io-index"
184 + checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
185 + dependencies = [
186 + "generic-array",
187 + "typenum",
188 + ]
189 +
123 190 [[package]]
124 191 name = "darling"
125 192 version = "0.23.0"
@@ -154,6 +221,22 @@
154 221 "syn",
155 222 ]
156 223
224 + [[package]]
225 + name = "data-encoding"
226 + version = "2.11.0"
227 + source = "registry+https://github.com/rust-lang/crates.io-index"
228 + checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8"
229 +
230 + [[package]]
231 + name = "digest"
232 + version = "0.10.7"
233 + source = "registry+https://github.com/rust-lang/crates.io-index"
234 + checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
235 + dependencies = [
236 + "block-buffer",
237 + "crypto-common",
238 + ]
239 +
157 240 [[package]]
158 241 name = "displaydoc"
159 242 version = "0.2.5"
@@ -223,6 +306,12 @@
223 306 source = "registry+https://github.com/rust-lang/crates.io-index"
224 307 checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
225 308
309 + [[package]]
310 + name = "futures-sink"
311 + version = "0.3.32"
312 + source = "registry+https://github.com/rust-lang/crates.io-index"
313 + checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893"
314 +
226 315 [[package]]
227 316 name = "futures-task"
228 317 version = "0.3.32"
@@ -236,11 +325,22 @@
236 325 checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6"
237 326 dependencies = [
238 327 "futures-core",
328 + "futures-sink",
239 329 "futures-task",
240 330 "pin-project-lite",
241 331 "slab",
242 332 ]
243 333
334 + [[package]]
335 + name = "generic-array"
336 + version = "0.14.7"
337 + source = "registry+https://github.com/rust-lang/crates.io-index"
338 + checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
339 + dependencies = [
340 + "typenum",
341 + "version_check",
342 + ]
343 +
244 344 [[package]]
245 345 name = "getrandom"
246 346 version = "0.2.17"
@@ -357,7 +457,7 @@
357 457 "tokio",
358 458 "tokio-rustls",
359 459 "tower-service",
360 - "webpki-roots",
460 + "webpki-roots 1.0.7",
361 461 ]
362 462
363 463 [[package]]
@@ -383,6 +483,30 @@
383 483 "tracing",
384 484 ]
385 485
486 + [[package]]
487 + name = "iana-time-zone"
488 + version = "0.1.65"
489 + source = "registry+https://github.com/rust-lang/crates.io-index"
490 + checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
491 + dependencies = [
492 + "android_system_properties",
493 + "core-foundation-sys",
494 + "iana-time-zone-haiku",
495 + "js-sys",
496 + "log",
497 + "wasm-bindgen",
498 + "windows-core",
499 + ]
500 +
501 + [[package]]
502 + name = "iana-time-zone-haiku"
503 + version = "0.1.2"
504 + source = "registry+https://github.com/rust-lang/crates.io-index"
505 + checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
506 + dependencies = [
507 + "cc",
508 + ]
509 +
386 510 [[package]]
387 511 name = "icu_collections"
388 512 version = "2.2.0"
@@ -613,6 +737,15 @@
613 737 "windows-sys 0.61.2",
614 738 ]
615 739
740 + [[package]]
741 + name = "num-traits"
742 + version = "0.2.19"
743 + source = "registry+https://github.com/rust-lang/crates.io-index"
744 + checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
745 + dependencies = [
746 + "autocfg",
747 + ]
748 +
616 749 [[package]]
617 750 name = "once_cell"
618 751 version = "1.21.4"
@@ -701,7 +834,7 @@
701 834 "rustc-hash",
702 835 "rustls",
703 836 "socket2",
704 - "thiserror",
837 + "thiserror 2.0.18",
705 838 "tokio",
706 839 "tracing",
707 840 "web-time",
@@ -716,13 +849,13 @@
716 849 "bytes",
717 850 "getrandom 0.3.4",
718 851 "lru-slab",
719 - "rand",
852 + "rand 0.9.4",
720 853 "ring",
721 854 "rustc-hash",
722 855 "rustls",
723 856 "rustls-pki-types",
724 857 "slab",
725 - "thiserror",
858 + "thiserror 2.0.18",
726 859 "tinyvec",
727 860 "tracing",
728 861 "web-time",
@@ -757,14 +890,35 @@
757 890 source = "registry+https://github.com/rust-lang/crates.io-index"
758 891 checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
759 892
893 + [[package]]
894 + name = "rand"
895 + version = "0.8.6"
896 + source = "registry+https://github.com/rust-lang/crates.io-index"
897 + checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
898 + dependencies = [
899 + "libc",
900 + "rand_chacha 0.3.1",
901 + "rand_core 0.6.4",
902 + ]
903 +
760 904 [[package]]
761 905 name = "rand"
762 906 version = "0.9.4"
763 907 source = "registry+https://github.com/rust-lang/crates.io-index"
764 908 checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
765 909 dependencies = [
766 - "rand_chacha",
767 - "rand_core",
910 + "rand_chacha 0.9.0",
911 + "rand_core 0.9.5",
912 + ]
913 +
914 + [[package]]
915 + name = "rand_chacha"
916 + version = "0.3.1"
917 + source = "registry+https://github.com/rust-lang/crates.io-index"
918 + checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
919 + dependencies = [
920 + "ppv-lite86",
921 + "rand_core 0.6.4",
768 922 ]
769 923
770 924 [[package]]
@@ -774,7 +928,16 @@
774 928 checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
775 929 dependencies = [
776 930 "ppv-lite86",
777 - "rand_core",
931 + "rand_core 0.9.5",
932 + ]
933 +
934 + [[package]]
935 + name = "rand_core"
936 + version = "0.6.4"
937 + source = "registry+https://github.com/rust-lang/crates.io-index"
938 + checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
939 + dependencies = [
940 + "getrandom 0.2.17",
778 941 ]
779 942
780 943 [[package]]
@@ -851,7 +1014,7 @@
851 1014 "wasm-bindgen",
852 1015 "wasm-bindgen-futures",
853 1016 "web-sys",
854 - "webpki-roots",
1017 + "webpki-roots 1.0.7",
855 1018 ]
856 1019
857 1020 [[package]]
@@ -939,12 +1102,15 @@
939 1102 version = "0.1.0"
940 1103 dependencies = [
941 1104 "anyhow",
1105 + "chrono",
942 1106 "crossterm",
1107 + "futures-util",
943 1108 "ratatui",
944 1109 "reqwest",
945 1110 "serde",
946 1111 "serde_json",
947 1112 "tokio",
1113 + "tokio-tungstenite",
948 1114 ]
949 1115
950 1116 [[package]]
@@ -1008,6 +1174,17 @@
1008 1174 "serde",
1009 1175 ]
1010 1176
1177 + [[package]]
1178 + name = "sha1"
1179 + version = "0.10.6"
1180 + source = "registry+https://github.com/rust-lang/crates.io-index"
1181 + checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
1182 + dependencies = [
1183 + "cfg-if",
1184 + "cpufeatures",
1185 + "digest",
1186 + ]
1187 +
1011 1188 [[package]]
1012 1189 name = "shlex"
1013 1190 version = "1.3.0"
@@ -1144,13 +1321,33 @@
1144 1321 "syn",
1145 1322 ]
1146 1323
1324 + [[package]]
1325 + name = "thiserror"
1326 + version = "1.0.69"
1327 + source = "registry+https://github.com/rust-lang/crates.io-index"
1328 + checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
1329 + dependencies = [
1330 + "thiserror-impl 1.0.69",
1331 + ]
1332 +
1147 1333 [[package]]
1148 1334 name = "thiserror"
1149 1335 version = "2.0.18"
1150 1336 source = "registry+https://github.com/rust-lang/crates.io-index"
1151 1337 checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
1152 1338 dependencies = [
1153 - "thiserror-impl",
1339 + "thiserror-impl 2.0.18",
1340 + ]
1341 +
1342 + [[package]]
1343 + name = "thiserror-impl"
1344 + version = "1.0.69"
1345 + source = "registry+https://github.com/rust-lang/crates.io-index"
1346 + checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
1347 + dependencies = [
1348 + "proc-macro2",
1349 + "quote",
1350 + "syn",
1154 1351 ]
1155 1352
1156 1353 [[package]]
@@ -1226,6 +1423,22 @@
1226 1423 "tokio",
1227 1424 ]
1228 1425
1426 + [[package]]
1427 + name = "tokio-tungstenite"
1428 + version = "0.24.0"
1429 + source = "registry+https://github.com/rust-lang/crates.io-index"
1430 + checksum = "edc5f74e248dc973e0dbb7b74c7e0d6fcc301c694ff50049504004ef4d0cdcd9"
1431 + dependencies = [
1432 + "futures-util",
1433 + "log",
1434 + "rustls",
1435 + "rustls-pki-types",
1436 + "tokio",
1437 + "tokio-rustls",
1438 + "tungstenite",
1439 + "webpki-roots 0.26.11",
1440 + ]
1441 +
1229 1442 [[package]]
1230 1443 name = "tower"
1231 1444 version = "0.5.3"
@@ -1296,6 +1509,32 @@
1296 1509 source = "registry+https://github.com/rust-lang/crates.io-index"
1297 1510 checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
1298 1511
1512 + [[package]]
1513 + name = "tungstenite"
1514 + version = "0.24.0"
1515 + source = "registry+https://github.com/rust-lang/crates.io-index"
1516 + checksum = "18e5b8366ee7a95b16d32197d0b2604b43a0be89dc5fac9f8e96ccafbaedda8a"
1517 + dependencies = [
1518 + "byteorder",
1519 + "bytes",
1520 + "data-encoding",
1521 + "http",
1522 + "httparse",
1523 + "log",
1524 + "rand 0.8.6",
1525 + "rustls",
1526 + "rustls-pki-types",
1527 + "sha1",
1528 + "thiserror 1.0.69",
1529 + "utf-8",
1530 + ]
1531 +
1532 + [[package]]
1533 + name = "typenum"
1534 + version = "1.20.1"
1535 + source = "registry+https://github.com/rust-lang/crates.io-index"
1536 + checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
1537 +
1299 1538 [[package]]
1300 1539 name = "unicode-ident"
1301 1540 version = "1.0.24"
@@ -1349,12 +1588,24 @@
1349 1588 "serde",
1350 1589 ]
1351 1590
1591 + [[package]]
1592 + name = "utf-8"
1593 + version = "0.7.6"
1594 + source = "registry+https://github.com/rust-lang/crates.io-index"
1595 + checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9"
1596 +
1352 1597 [[package]]
1353 1598 name = "utf8_iter"
1354 1599 version = "1.0.4"
1355 1600 source = "registry+https://github.com/rust-lang/crates.io-index"
1356 1601 checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
1357 1602
1603 + [[package]]
1604 + name = "version_check"
1605 + version = "0.9.5"
1606 + source = "registry+https://github.com/rust-lang/crates.io-index"
1607 + checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
1608 +
1358 1609 [[package]]
1359 1610 name = "want"
1360 1611 version = "0.3.1"
@@ -1454,6 +1705,15 @@
1454 1705 "wasm-bindgen",
1455 1706 ]
1456 1707
1708 + [[package]]
1709 + name = "webpki-roots"
1710 + version = "0.26.11"
1711 + source = "registry+https://github.com/rust-lang/crates.io-index"
1712 + checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
1713 + dependencies = [
1714 + "webpki-roots 1.0.7",
1715 + ]
1716 +
1457 1717 [[package]]
1458 1718 name = "webpki-roots"
1459 1719 version = "1.0.7"
@@ -1485,12 +1745,65 @@
1485 1745 source = "registry+https://github.com/rust-lang/crates.io-index"
1486 1746 checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
1487 1747
1748 + [[package]]
1749 + name = "windows-core"
1750 + version = "0.62.2"
1751 + source = "registry+https://github.com/rust-lang/crates.io-index"
1752 + checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
1753 + dependencies = [
1754 + "windows-implement",
1755 + "windows-interface",
1756 + "windows-link",
1757 + "windows-result",
1758 + "windows-strings",
1759 + ]
1760 +
1761 + [[package]]
1762 + name = "windows-implement"
1763 + version = "0.60.2"
1764 + source = "registry+https://github.com/rust-lang/crates.io-index"
1765 + checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
1766 + dependencies = [
1767 + "proc-macro2",
1768 + "quote",
1769 + "syn",
1770 + ]
1771 +
1772 + [[package]]
1773 + name = "windows-interface"
1774 + version = "0.59.3"
1775 + source = "registry+https://github.com/rust-lang/crates.io-index"
1776 + checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
1777 + dependencies = [
1778 + "proc-macro2",
1779 + "quote",
1780 + "syn",
1781 + ]
1782 +
Lines truncated
M sando/tui/Cargo.toml +4 -1
@@ -11,8 +11,11 @@
11 11 [dependencies]
12 12 ratatui = "0.29"
13 13 crossterm = "0.28"
14 - tokio = { version = "1.50.0", features = ["macros", "rt-multi-thread", "net", "signal"] }
14 + tokio = { version = "1.50.0", features = ["macros", "rt-multi-thread", "net", "signal", "sync", "time"] }
15 + tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-webpki-roots"] }
16 + futures-util = { version = "0.3", default-features = false }
15 17 reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
16 18 serde = { version = "1.0.228", features = ["derive"] }
17 19 serde_json = "1"
18 20 anyhow = "1.0.102"
21 + chrono = { version = "0.4", default-features = false, features = ["clock", "serde"] }
M sando/daemon/src/backup.rs +202 -22
@@ -1,9 +1,11 @@
1 1 //! Fetch the prod backup that `migration_dry_run` runs against.
2 2 //!
3 - //! Sources supported in v0:
4 - //! - `file:///abs/path/to/dump.sql.gz` — local copy. Used for localhost dev.
5 - //! - `rsync://host/module/path` — shells out to `rsync`. Used when MM
6 - //! pulls from an astra/Hetzner replica.
3 + //! Sources supported:
4 + //! - `file:///abs/path/to/dump.sql.gz` — local copy (dev).
5 + //! - `rsync://host/module/path` — rsync daemon protocol.
6 + //! - `ssh://user@host[:port]/path/file.sql.gz` — rsync-over-ssh. Used to pull
7 + //! prod backups from
8 + //! `backup-puller@alpha-west-1`.
7 9 //!
8 10 //! The fetch is command-driven: the operator triggers it via /backup/fetch, it
9 11 //! is not implicit in promote. That keeps the slowest, most failure-prone step
@@ -11,7 +13,7 @@
11 13
12 14 use crate::config::Config;
13 15 use crate::topology::Topology;
14 - use anyhow::{Context, Result};
16 + use anyhow::{Context, Result, bail};
15 17 use chrono::Utc;
16 18 use sqlx::SqlitePool;
17 19 use std::path::Path;
@@ -25,6 +27,61 @@
25 27 pub byte_size: Option<i64>,
26 28 }
27 29
30 + /// Parsed `backup.source` URL. Owned strings so the parsed form outlives the
31 + /// (possibly transient) URL we read from config.
32 + #[derive(Debug, Clone, PartialEq, Eq)]
33 + pub(crate) enum BackupSource {
34 + /// Local file copy. Path follows the `file://` prefix.
35 + File { path: String },
36 + /// rsync daemon protocol. Full URL stays intact (rsync handles it).
37 + RsyncDaemon { url: String },
38 + /// rsync-over-ssh. Port is optional.
39 + Ssh {
40 + user_host: String,
41 + port: Option<u16>,
42 + path: String,
43 + },
44 + }
45 +
46 + /// Parse a `backup.source` URL into a `BackupSource`. Rejects unsupported
47 + /// schemes and malformed `ssh://` URLs (no path part).
48 + pub(crate) fn parse_source(s: &str) -> Result<BackupSource> {
49 + if let Some(rest) = s.strip_prefix("file://") {
50 + if rest.is_empty() {
51 + bail!("file:// URL is missing a path: {s}");
52 + }
53 + return Ok(BackupSource::File { path: rest.into() });
54 + }
55 + if s.starts_with("rsync://") {
56 + return Ok(BackupSource::RsyncDaemon { url: s.into() });
57 + }
58 + if let Some(rest) = s.strip_prefix("ssh://") {
59 + let (user_host_port, path_rest) = rest
60 + .split_once('/')
61 + .with_context(|| format!("ssh:// URL missing path: {s}"))?;
62 + if user_host_port.is_empty() {
63 + bail!("ssh:// URL missing user@host: {s}");
64 + }
65 + let path = format!("/{path_rest}");
66 + let (user_host, port) = match user_host_port.rsplit_once(':') {
67 + Some((uh, p)) => {
68 + // Heuristic: trailing `:digits` after the final `:` is the port.
69 + // Anything else (IPv6 literal, etc.) gets left alone.
70 + match p.parse::<u16>() {
71 + Ok(n) => (uh.to_string(), Some(n)),
72 + Err(_) => (user_host_port.to_string(), None),
73 + }
74 + }
75 + None => (user_host_port.to_string(), None),
76 + };
77 + if user_host.is_empty() {
78 + bail!("ssh:// URL has empty host (port {:?})", port);
79 + }
80 + return Ok(BackupSource::Ssh { user_host, port, path });
81 + }
82 + bail!("unsupported backup source scheme: {s}");
83 + }
84 +
28 85 pub async fn fetch(
29 86 pool: &SqlitePool,
30 87 _cfg: &Arc<Config>,
@@ -37,23 +94,45 @@
37 94 tokio::fs::create_dir_all(parent).await?;
38 95 }
39 96
40 - if let Some(rest) = source.strip_prefix("file://") {
41 - tokio::fs::copy(rest, &local_path)
42 - .await
43 - .with_context(|| format!("copy {rest} -> {local_path}"))?;
44 - } else if source.starts_with("rsync://") {
45 - let out = Command::new("rsync")
46 - .args(["-az", "--inplace", &source, &local_path])
47 - .output()
48 - .await
49 - .context("spawning rsync")?;
50 - anyhow::ensure!(
51 - out.status.success(),
52 - "rsync failed: {}",
53 - String::from_utf8_lossy(&out.stderr),
54 - );
55 - } else {
56 - anyhow::bail!("unsupported backup source scheme: {source}");
97 + let parsed = parse_source(&source)?;
98 + match parsed {
99 + BackupSource::File { path } => {
100 + tokio::fs::copy(&path, &local_path)
101 + .await
102 + .with_context(|| format!("copy {path} -> {local_path}"))?;
103 + }
104 + BackupSource::RsyncDaemon { url } => {
105 + let out = Command::new("rsync")
106 + .args(["-az", "--inplace", &url, &local_path])
107 + .output()
108 + .await
109 + .context("spawning rsync")?;
110 + anyhow::ensure!(
111 + out.status.success(),
112 + "rsync (daemon) failed: {}",
113 + String::from_utf8_lossy(&out.stderr),
114 + );
115 + }
116 + BackupSource::Ssh { user_host, port, path } => {
117 + let ssh_cmd = match port {
118 + Some(p) => format!("ssh -p {p} -o BatchMode=yes -o StrictHostKeyChecking=accept-new"),
119 + None => "ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new".into(),
120 + };
121 + let remote = format!("{user_host}:{path}");
122 + let out = Command::new("rsync")
123 + .args(["-a", "--partial"])
124 + .arg("-e").arg(&ssh_cmd)
125 + .arg(&remote)
126 + .arg(&local_path)
127 + .output()
128 + .await
129 + .context("spawning rsync")?;
130 + anyhow::ensure!(
131 + out.status.success(),
132 + "rsync (ssh) failed: {}",
133 + String::from_utf8_lossy(&out.stderr),
134 + );
135 + }
57 136 }
58 137
59 138 let meta = tokio::fs::metadata(&local_path).await?;
@@ -69,5 +148,106 @@
69 148 .execute(pool)
70 149 .await?;
71 150
151 + // Retention: prune rows fetched more than 30 days ago. The on-disk file
152 + // is overwritten each fetch (single `local_path`), so old rows reference
153 + // a path that no longer exists — keep the table from growing for no
154 + // good reason.
155 + sqlx::query(
156 + "DELETE FROM backups WHERE fetched_at < datetime('now', '-30 days')",
157 + )
158 + .execute(pool)
159 + .await?;
160 +
72 161 Ok(FetchedBackup { source, local_path, byte_size: Some(size) })
73 162 }
163 +
164 + #[cfg(test)]
165 + mod tests {
166 + use super::*;
167 +
168 + #[test]
169 + fn parses_file_url() {
170 + let s = parse_source("file:///opt/backups/latest.sql.gz").unwrap();
171 + assert_eq!(s, BackupSource::File { path: "/opt/backups/latest.sql.gz".into() });
172 + }
173 +
174 + #[test]
175 + fn file_url_without_path_errors() {
176 + assert!(parse_source("file://").is_err());
177 + }
178 +
179 + #[test]
180 + fn parses_rsync_daemon_url() {
181 + let s = parse_source("rsync://astra/mnw/latest.sql.gz").unwrap();
182 + assert_eq!(s, BackupSource::RsyncDaemon { url: "rsync://astra/mnw/latest.sql.gz".into() });
183 + }
184 +
185 + #[test]
186 + fn parses_ssh_url_with_port() {
187 + let s = parse_source("ssh://backup-puller@alpha-west-1:2200/latest.sql.gz").unwrap();
188 + assert_eq!(
189 + s,
190 + BackupSource::Ssh {
191 + user_host: "backup-puller@alpha-west-1".into(),
192 + port: Some(2200),
193 + path: "/latest.sql.gz".into(),
194 + }
195 + );
196 + }
197 +
198 + #[test]
199 + fn parses_ssh_url_without_port() {
200 + let s = parse_source("ssh://max@astra/opt/backups/mnw/latest.sql.gz").unwrap();
201 + assert_eq!(
202 + s,
203 + BackupSource::Ssh {
204 + user_host: "max@astra".into(),
205 + port: None,
206 + path: "/opt/backups/mnw/latest.sql.gz".into(),
207 + }
208 + );
209 + }
210 +
211 + #[test]
212 + fn ssh_url_without_path_errors() {
213 + // `split_once('/')` — `ssh://user@host` has no `/` after the scheme.
214 + assert!(parse_source("ssh://backup-puller@alpha-west-1").is_err());
215 + }
216 +
217 + #[test]
218 + fn ssh_url_without_user_host_errors() {
219 + // Empty user@host: `ssh:///foo`. Caught by the empty-prefix check.
220 + assert!(parse_source("ssh:///latest.sql.gz").is_err());
221 + }
222 +
223 + #[test]
224 + fn ssh_url_with_non_numeric_after_colon_treats_as_part_of_host() {
225 + // `host:notaport` should NOT parse `notaport` as a port. Leave the
226 + // colon part of user_host; libssh/rsync will reject if truly wrong.
227 + let s = parse_source("ssh://user@host:notaport/path").unwrap();
228 + assert_eq!(
229 + s,
230 + BackupSource::Ssh {
231 + user_host: "user@host:notaport".into(),
232 + port: None,
233 + path: "/path".into(),
234 + }
235 + );
236 + }
237 +
238 + #[test]
239 + fn rejects_unknown_scheme() {
240 + assert!(parse_source("ftp://example.com/file").is_err());
241 + assert!(parse_source("just-a-path.sql.gz").is_err());
242 + assert!(parse_source("").is_err());
243 + }
244 +
245 + #[test]
246 + fn ssh_url_preserves_multi_segment_path() {
247 + let s = parse_source("ssh://a@b:22/opt/foo/bar/baz.sql.gz").unwrap();
248 + match s {
249 + BackupSource::Ssh { path, .. } => assert_eq!(path, "/opt/foo/bar/baz.sql.gz"),
250 + _ => panic!("wrong variant"),
251 + }
252 + }
253 + }
M sando/daemon/src/build.rs +73 -15
@@ -21,7 +21,10 @@
21 21 pub version: String,
22 22 pub git_sha: String,
23 23 pub worktree: PathBuf,
24 - pub binary_path: PathBuf,
24 + /// One entry per `cfg.bin_names` in declared order. First is the primary
25 + /// (referenced by the systemd unit's ExecStart). Paths are inside the
26 + /// worktree's `target/release/`.
27 + pub binary_paths: Vec<PathBuf>,
25 28 }
26 29
27 30 pub async fn run(
@@ -29,6 +32,7 @@
29 32 cfg: Arc<Config>,
30 33 topo: Arc<Topology>,
31 34 sha: String,
35 + events: crate::events::EventTx,
32 36 ) -> Result<BuildArtifact> {
33 37 let worktree = cfg.workdir.join(&sha);
34 38 let bare = PathBuf::from(&topo.repo.bare_path);
@@ -38,20 +42,47 @@
38 42 let version = read_pkg_version(&server_dir.join("Cargo.toml")).await
39 43 .with_context(|| format!("reading version from {}/Cargo.toml", server_dir.display()))?;
40 44
41 - tracing::info!(sha = %sha, version = %version, dir = %server_dir.display(), "cargo build --release start");
42 - let started = std::time::Instant::now();
43 - let out = Command::new("cargo")
45 + // sqlx compile-time query checking needs a live DB with the current schema.
46 + // We point cargo at the scratch DB and prep it (drop public, re-migrate)
47 + // before invoking cargo build. The same DB is reset again by
48 + // `migration_dry_run` later if it runs as a gate.
49 + let mut cargo_cmd = Command::new("cargo");
50 + cargo_cmd
44 51 .arg("build")
45 52 .arg("--release")
46 53 .current_dir(&server_dir)
54 + .kill_on_drop(true);
55 + if let Some(scratch_url) = cfg.scratch_db_url.as_deref() {
56 + tracing::info!(sha = %sha, "preparing scratch DB schema for sqlx compile-time checks");
57 + crate::gates::reset_scratch(scratch_url).await
58 + .context("scratch DB reset before build")?;
59 + crate::gates::run_migrator(scratch_url, &server_dir.join("migrations")).await
60 + .context("applying MNW migrations to scratch DB before build")?;
61 + cargo_cmd.env("DATABASE_URL", scratch_url);
62 + } else {
63 + tracing::warn!("scratch_db_url unset; sqlx will fall back to offline mode and may fail");
64 + }
65 +
66 + tracing::info!(sha = %sha, version = %version, dir = %server_dir.display(), "cargo build --release start");
67 + crate::events::emit(&events, crate::events::Event::BuildStart {
68 + sha: sha.clone(), version: version.clone(),
69 + });
70 + let started = std::time::Instant::now();
71 + let out = cargo_cmd
47 72 .output()
48 73 .await
49 74 .context("spawning cargo build")?;
50 75 let elapsed_s = started.elapsed().as_secs();
51 76 if !out.status.success() {
52 77 tracing::error!(sha = %sha, version = %version, elapsed_s, "cargo build --release failed");
78 + crate::events::emit(&events, crate::events::Event::BuildFailed {
79 + sha: sha.clone(), version: version.clone(), elapsed_s,
80 + });
53 81 } else {
54 82 tracing::info!(sha = %sha, version = %version, elapsed_s, "cargo build --release ok");
83 + crate::events::emit(&events, crate::events::Event::BuildOk {
84 + sha: sha.clone(), version: version.clone(), elapsed_s,
85 + });
55 86 }
56 87 anyhow::ensure!(
57 88 out.status.success(),
@@ -59,12 +90,16 @@
59 90 tail(&out.stderr, 4_000),
60 91 );
61 92
62 - let binary_path = server_dir.join("target/release/server");
63 - anyhow::ensure!(
64 - binary_path.exists(),
65 - "expected binary at {} after build",
66 - binary_path.display(),
67 - );
93 + let release_dir = server_dir.join("target/release");
94 + let mut binary_paths = Vec::with_capacity(cfg.bin_names.len());
95 + for name in &cfg.bin_names {
96 + let p = release_dir.join(name);
97 + anyhow::ensure!(p.exists(), "expected binary at {} after build", p.display());
98 + binary_paths.push(p);
99 + }
100 + // Primary binary path is the one we record in `versions.artifact_path`
101 + // (everything downstream — promote, rollback — looks it up by version).
102 + let primary = binary_paths[0].clone();
68 103
69 104 sqlx::query(
70 105 "INSERT OR IGNORE INTO versions (version, git_sha, built_at, artifact_path)
@@ -73,11 +108,11 @@
73 108 .bind(&version)
74 109 .bind(&sha)
75 110 .bind(Utc::now().to_rfc3339())
76 - .bind(binary_path.to_string_lossy().as_ref())
111 + .bind(primary.to_string_lossy().as_ref())
77 112 .execute(&pool)
78 113 .await?;
79 114
80 - Ok(BuildArtifact { version, git_sha: sha, worktree, binary_path })
115 + Ok(BuildArtifact { version, git_sha: sha, worktree, binary_paths })
81 116 }
82 117
83 118 /// Full MM-tier pipeline: build, deploy the binary into MM's release_root,
@@ -88,14 +123,36 @@
88 123 cfg: Arc<Config>,
89 124 topo: Arc<Topology>,
90 125 sha: String,
126 + events: crate::events::EventTx,
91 127 ) -> Result<()> {
92 - let art = run(pool.clone(), cfg.clone(), topo.clone(), sha).await?;
128 + let art = run(pool.clone(), cfg.clone(), topo.clone(), sha, events.clone()).await?;
93 129
94 130 // Stage the binary in MM's release_root so future gates and the MM
95 131 // self-deploy point at a stable path, not the worktree's target/.
96 132 let mm_release_root = &cfg.release_root;
97 - let staged = deploy::deploy_local(mm_release_root, &art.version, &art.binary_path).await?;
98 - let staged_bin = staged.join("server");
133 + let staged = deploy::deploy_local(mm_release_root, &art.version, &art.binary_paths).await?;
134 +
135 + // Bring error-pages alongside the binaries so the deploy rsync ships the
136 + // static HTML to every node. Caddy on each node references
137 + // <release_root>/current/error-pages/. Skipped silently if the worktree
138 + // doesn't have them (older shas, or non-MNW projects using this daemon).
139 + let error_pages_src = art.worktree.join("server/deploy/error-pages");
140 + if error_pages_src.exists() {
141 + let out = Command::new("cp")
142 + .arg("-a")
143 + .arg(&error_pages_src)
144 + .arg(staged.join("error-pages"))
145 + .output()
146 + .await
147 + .context("spawning cp for error-pages")?;
148 + anyhow::ensure!(
149 + out.status.success(),
150 + "copying error-pages into staged dir: {}",
151 + String::from_utf8_lossy(&out.stderr),
152 + );
153 + }
154 +
155 + let staged_bin = staged.join(cfg.primary_bin());
99 156 sqlx::query("UPDATE versions SET artifact_path = ? WHERE version = ?")
100 157 .bind(staged_bin.to_string_lossy().as_ref())
101 158 .bind(&art.version)
@@ -112,6 +169,7 @@
112 169 tier: "mm".to_string(),
113 170 version: art.version.clone(),
114 171 worktree: art.worktree.clone(),
172 + events: events.clone(),
115 173 };
116 174 let ok = gates::run_all(&ctx, &mm.gates).await?;
117 175
M sando/daemon/src/config.rs +13
@@ -16,9 +16,22 @@
16 16 /// you care about.
17 17 #[serde(default)]
18 18 pub scratch_db_url: Option<String>,
19 + /// Names of cargo bin targets the server crate produces (files under
20 + /// `target/release/`). First entry is the primary unit (referenced from
21 + /// the systemd unit's ExecStart). Defaults to `["server"]`; MNW ships
22 + /// `["makenotwork", "mnw-admin"]`.
23 + #[serde(default = "default_bin_names")]
24 + pub bin_names: Vec<String>,
19 25 }
20 26
27 + fn default_bin_names() -> Vec<String> { vec!["server".into()] }
28 +
21 29 impl Config {
30 + /// Primary binary — the one the systemd unit's ExecStart points at.
31 + pub fn primary_bin(&self) -> &str {
32 + self.bin_names.first().map(|s| s.as_str()).unwrap_or("server")
33 + }
34 +
22 35 pub fn load() -> Result<Self> {
23 36 let path = std::env::var("SANDO_CONFIG").unwrap_or_else(|_| "sando-daemon.toml".into());
24 37 let raw = std::fs::read_to_string(&path)
M sando/daemon/src/deploy.rs +395 -31
@@ -1,40 +1,58 @@
1 1 //! Atomic symlink-swap deploys.
2 2 //!
3 - //! Layout on every target (MM, A nodes, B nodes, ...):
3 + //! Layout on every target (local host, A nodes, B nodes, ...):
4 4 //!
5 5 //! <release_root>/
6 6 //! releases/
7 7 //! 0.8.1/
8 - //! server <- the binary
8 + //! <bin_name>
9 9 //! 0.8.2/
10 - //! server
10 + //! <bin_name>
11 11 //! current -> releases/0.8.2
12 12 //!
13 - //! `ln -sfn` makes the swap atomic on Linux. systemd units should point at
14 - //! `<release_root>/current/server` so a swap + reload picks up the new binary
15 - //! without a window where the unit references a missing path.
13 + //! `ln -sfn` swaps the symlink. systemd units point at
14 + //! `<release_root>/current/<bin_name>` so reload-or-restart picks up the new
15 + //! binary without ever pointing at a missing path.
16 16 //!
17 - //! v0 only implements local deploys (used for MM and for localhost-dev
18 - //! "remote" nodes whose ssh_target is `local`). Real SSH/rsync deploys are
19 - //! follow-up work — see the `remote_deploy_stub` branch.
17 + //! For nodes with `ssh_target` set to anything other than `"local"`, deploy
18 + //! goes via rsync + ssh; the bootstrap (creating release_root, installing the
19 + //! service unit, granting sudo for systemctl) is out of scope here — it
20 + //! happens once per node, not per deploy.
20 21
21 22 use crate::topology::Node;
22 23 use anyhow::{Context, Result};
23 24 use std::path::{Path, PathBuf};
24 25 use tokio::process::Command;
25 26
26 - pub async fn deploy_local(release_root: &Path, version: &str, binary: &Path) -> Result<PathBuf> {
27 + /// SSH options used everywhere we shell out to ssh — fail fast, no prompts.
28 + const SSH_FLAGS: &[&str] = &[
29 + "-o", "BatchMode=yes",
30 + "-o", "ConnectTimeout=10",
31 + "-o", "StrictHostKeyChecking=accept-new",
32 + ];
33 +
34 + /// Keep this many release dirs per node; older ones get gc'd after a
35 + /// successful deploy. Fixed for now; promote to config if the constant ever
36 + /// needs to vary by tier.
37 + const RELEASES_TO_KEEP: usize = 5;
38 +
39 + pub async fn deploy_local(
40 + release_root: &Path,
41 + version: &str,
42 + binaries: &[PathBuf],
43 + ) -> Result<PathBuf> {
27 44 let release_dir = release_root.join("releases").join(version);
28 45 tokio::fs::create_dir_all(&release_dir).await?;
29 - let dest = release_dir.join("server");
30 - tokio::fs::copy(binary, &dest)
31 - .await
32 - .with_context(|| format!("copy {} -> {}", binary.display(), dest.display()))?;
46 + for binary in binaries {
47 + let name = binary.file_name()
48 + .context("binary path has no file name")?;
49 + let dest = release_dir.join(name);
50 + tokio::fs::copy(binary, &dest)
51 + .await
52 + .with_context(|| format!("copy {} -> {}", binary.display(), dest.display()))?;
53 + }
33 54
34 55 let current = release_root.join("current");
35 - // ln -sfn is atomic on Linux; on macOS the dev path is non-prod so the
36 - // race is irrelevant. We shell out rather than using std::os::unix::fs
37 - // symlink + rename because the rename-over-symlink pattern is platform-fussy.
38 56 let target = format!("releases/{version}");
39 57 let out = Command::new("ln")
40 58 .args(["-sfn", &target])
@@ -46,26 +64,372 @@
46 64 "symlink swap failed: {}",
47 65 String::from_utf8_lossy(&out.stderr),
48 66 );
67 +
68 + if let Err(e) = gc_local_releases(release_root).await {
69 + tracing::warn!(error = %e, "local release GC failed (non-fatal)");
70 + }
49 71 Ok(release_dir)
50 72 }
51 73
52 - pub async fn deploy_node(node: &Node, version: &str, binary: &Path) -> Result<PathBuf> {
74 + /// Deploy `staged_release_dir` (a directory built on the Sando host by
75 + /// `deploy_local`) to `node`. For `ssh_target=local`, this is just symlink
76 + /// swap + restart; for remote nodes, we rsync the whole dir.
77 + ///
78 + /// `primary_bin` is only used for logging — every file present in the staged
79 + /// dir gets shipped.
80 + pub async fn deploy_node(
81 + node: &Node,
82 + version: &str,
83 + staged_release_dir: &Path,
84 + primary_bin: &str,
85 + ) -> Result<PathBuf> {
53 86 if node.ssh_target == "local" || node.ssh_target.is_empty() {
54 - return deploy_local(Path::new(&node.release_root), version, binary).await;
87 + // Local deploy already happened when we staged on the Sando host.
88 + // Just re-point `current` at the staged dir.
89 + return reset_local_current(Path::new(&node.release_root), version).await;
55 90 }
56 - remote_deploy_stub(node, version, binary).await
91 + deploy_remote(node, version, staged_release_dir, primary_bin).await
57 92 }
58 93
59 - async fn remote_deploy_stub(node: &Node, version: &str, _binary: &Path) -> Result<PathBuf> {
60 - // Real implementation: rsync the binary to <ssh_target>:<release_root>/releases/<version>/server,
61 - // then ssh <ssh_target> "ln -sfn releases/<version> current && systemctl reload-or-restart <unit>".
62 - // Wiring this up needs a story for systemd unit naming and ssh key/auth conventions; deferring
63 - // until the localhost smoke loop is settled and we know which knobs matter.
64 - anyhow::bail!(
65 - "remote deploy not yet implemented (node {} -> {}); use ssh_target=local for dev",
66 - node.name,
67 - node.ssh_target,
94 + async fn reset_local_current(release_root: &Path, version: &str) -> Result<PathBuf> {
95 + let current = release_root.join("current");
96 + let target = format!("releases/{version}");
97 + let out = Command::new("ln")
98 + .args(["-sfn", &target])
99 + .arg(&current)
100 + .output()
101 + .await?;
102 + anyhow::ensure!(
103 + out.status.success(),
104 + "symlink swap failed: {}",
105 + String::from_utf8_lossy(&out.stderr),
68 106 );
69 - #[allow(unreachable_code)]
70 - Ok(PathBuf::from(&node.release_root).join("releases").join(version))
107 + Ok(release_root.join("releases").join(version))
108 + }
109 +
110 + async fn deploy_remote(
111 + node: &Node,
112 + version: &str,
113 + staged_release_dir: &Path,
114 + primary_bin: &str,
115 + ) -> Result<PathBuf> {
116 + let release_root = &node.release_root;
117 + let ssh_target = &node.ssh_target;
118 + let service = &node.service_name;
119 + let release_dir = format!("{release_root}/releases/{version}");
120 +
121 + tracing::info!(node = %node.name, version, "deploy: mkdir release dir");
122 + ssh(ssh_target, &format!("set -e; mkdir -p {q}", q = sh_quote(&release_dir)))
123 + .await
124 + .context("creating remote release dir")?;
125 +
126 + tracing::info!(node = %node.name, version, primary = %primary_bin, "deploy: rsync release dir");
127 + // Rsync the whole staged dir (all binaries + any sibling artifacts like
128 + // error-pages). Trailing slash on source = contents of dir, not the dir
129 + // itself. --chmod ensures binaries land executable; the regular-file
130 + // mask leaves data files at 0644.
131 + let rsync_src = format!("{}/", staged_release_dir.display());
132 + let rsync_dest = format!("{ssh_target}:{release_dir}/");
133 + let mut rsync = Command::new("rsync");
134 + rsync
135 + .arg("-az")
136 + .arg("--partial")
137 + .arg("--chmod=F0755,D0755")
138 + .arg("-e")
139 + .arg(format!(
140 + "ssh {}",
141 + SSH_FLAGS.iter().map(|s| s.to_string()).collect::<Vec<_>>().join(" ")
142 + ))
143 + .arg(&rsync_src)
144 + .arg(&rsync_dest);
145 + let out = rsync.output().await.context("spawning rsync")?;
146 + anyhow::ensure!(
147 + out.status.success(),
148 + "rsync failed (current symlink left intact): {}",
149 + String::from_utf8_lossy(&out.stderr),
150 + );
151 +
152 + tracing::info!(node = %node.name, version, "deploy: symlink swap + service reload");
153 + // Symlink swap is atomic via `mv -T` of a freshly-created symlink over
154 + // the old one (the rename(2) is the atomic step; `ln -sfn` does
155 + // unlink+symlink which has a window).
156 + let swap_and_restart = format!(
157 + "set -e; \
158 + cd {root}; \
159 + ln -sfn releases/{ver} current.new; \
160 + mv -Tf current.new current; \
161 + sudo /bin/systemctl reload-or-restart {svc}",
162 + root = sh_quote(release_root),
163 + ver = sh_quote(version),
164 + svc = sh_quote(service),
165 + );
166 + ssh(ssh_target, &swap_and_restart)
167 + .await
168 + .context("symlink swap + systemctl reload-or-restart")?;
169 +
170 + if let Err(e) = gc_remote_releases(ssh_target, release_root).await {
171 + tracing::warn!(error = %e, "remote release GC failed (non-fatal)");
172 + }
173 +
174 + Ok(PathBuf::from(release_root).join("releases").join(version))
175 + }
176 +
177 + async fn ssh(target: &str, script: &str) -> Result<()> {
178 + let mut cmd = Command::new("ssh");
179 + cmd.args(SSH_FLAGS).arg(target).arg(script);
180 + let out = cmd.output().await.context("spawning ssh")?;
181 + anyhow::ensure!(
182 + out.status.success(),
183 + "ssh {target} failed: {}",
184 + String::from_utf8_lossy(&out.stderr),
185 + );
186 + Ok(())
187 + }
188 +
189 + async fn gc_local_releases(release_root: &Path) -> Result<()> {
190 + let releases = release_root.join("releases");
191 + if !releases.exists() {
192 + return Ok(());
193 + }
194 + let mut entries = Vec::new();
195 + let mut rd = tokio::fs::read_dir(&releases).await?;
196 + while let Some(entry) = rd.next_entry().await? {
197 + if !entry.file_type().await?.is_dir() {
198 + continue;
199 + }
200 + let meta = entry.metadata().await?;
201 + entries.push((entry.path(), meta.modified()?));
202 + }
203 + entries.sort_by(|a, b| b.1.cmp(&a.1));
204 + for (path, _) in entries.into_iter().skip(RELEASES_TO_KEEP) {
205 + if let Err(e) = tokio::fs::remove_dir_all(&path).await {
206 + tracing::warn!(path = %path.display(), error = %e, "gc: rm failed");
207 + } else {
208 + tracing::debug!(path = %path.display(), "gc: removed old release");
209 + }
210 + }
211 + Ok(())
212 + }
213 +
214 + async fn gc_remote_releases(ssh_target: &str, release_root: &str) -> Result<()> {
215 + // `ls -t` orders by mtime desc. Skip the first N, rm the rest. `xargs -r`
216 + // is a no-op when stdin is empty (avoids `rm` complaining).
217 + let script = format!(
218 + "set -e; cd {root}/releases 2>/dev/null || exit 0; \
219 + ls -1t | tail -n +{keep_plus_one} | xargs -r -I{{}} rm -rf -- {{}}",
220 + root = sh_quote(release_root),
221 + keep_plus_one = RELEASES_TO_KEEP + 1,
222 + );
223 + ssh(ssh_target, &script).await
224 + }
225 +
226 + /// Single-quote a string for safe inclusion in a /bin/sh command, escaping
227 + /// any single quote inside. Not bulletproof for adversarial input, but every
228 + /// path here comes from our own config files.
229 + fn sh_quote(s: &str) -> String {
230 + let escaped = s.replace('\'', r"'\''");
231 + format!("'{escaped}'")
232 + }
233 +
234 + #[cfg(test)]
235 + mod tests {
236 + use super::*;
237 + use std::time::SystemTime;
238 +
239 + #[test]
240 + fn sh_quote_no_quote() {
241 + assert_eq!(sh_quote("hello"), "'hello'");
242 + assert_eq!(sh_quote("/opt/mnw/releases/0.8.12"), "'/opt/mnw/releases/0.8.12'");
243 + }
244 +
245 + #[test]
246 + fn sh_quote_with_quote() {
247 + // The string `it's` becomes `'it'\''s'` — close, escape, open.
248 + assert_eq!(sh_quote("it's"), r"'it'\''s'");
249 + }
250 +
251 + #[tokio::test]
252 + async fn deploy_local_copies_multiple_binaries_and_swaps_symlink() {
253 + let tmp = tempfile::tempdir().unwrap();
254 + let root = tmp.path();
255 +
256 + // Source binaries (worktree's target/release/)
257 + let src_dir = root.join("src");
258 + tokio::fs::create_dir_all(&src_dir).await.unwrap();
259 + let primary = src_dir.join("makenotwork");
260 + let admin = src_dir.join("mnw-admin");
261 + tokio::fs::write(&primary, b"PRIMARY").await.unwrap();
262 + tokio::fs::write(&admin, b"ADMIN").await.unwrap();
263 +
264 + // Release root (where staged versions live)
265 + let release_root = root.join("releases-root");
266 + tokio::fs::create_dir_all(&release_root).await.unwrap();
267 +
268 + let staged = deploy_local(
269 + &release_root,
270 + "0.8.12",
271 + &[primary.clone(), admin.clone()],
272 + )
273 + .await
274 + .expect("deploy_local should succeed");
275 +
276 + assert_eq!(staged, release_root.join("releases").join("0.8.12"));
277 + assert_eq!(tokio::fs::read(staged.join("makenotwork")).await.unwrap(), b"PRIMARY");
278 + assert_eq!(tokio::fs::read(staged.join("mnw-admin")).await.unwrap(), b"ADMIN");
279 +
280 + // current symlink should resolve to staged
281 + let current = release_root.join("current");
282 + let target = tokio::fs::read_link(&current).await.unwrap();
283 + assert_eq!(target.to_string_lossy(), "releases/0.8.12");
284 + // And reading through `current/` should give the new content.
285 + let via_current = tokio::fs::read(current.join("makenotwork")).await.unwrap();
286 + assert_eq!(via_current, b"PRIMARY");
287 + }
288 +
289 + #[tokio::test]
290 + async fn deploy_local_second_release_swaps_symlink_and_keeps_old_dir() {
291 + let tmp = tempfile::tempdir().unwrap();
292 + let root = tmp.path();
293 + let src_dir = root.join("src");
294 + tokio::fs::create_dir_all(&src_dir).await.unwrap();
295 + let bin = src_dir.join("server");
296 + tokio::fs::write(&bin, b"V1").await.unwrap();
297 +
298 + let release_root = root.join("rr");
299 + tokio::fs::create_dir_all(&release_root).await.unwrap();
300 +
301 + deploy_local(&release_root, "0.1.0", &[bin.clone()]).await.unwrap();
302 + // Rewrite source then deploy 0.2.0.
303 + tokio::fs::write(&bin, b"V2").await.unwrap();
304 + deploy_local(&release_root, "0.2.0", &[bin.clone()]).await.unwrap();
305 +
306 + // Both versions present on disk.
307 + assert!(release_root.join("releases/0.1.0/server").exists());
308 + assert!(release_root.join("releases/0.2.0/server").exists());
309 + // current points at the new one.
310 + let target = tokio::fs::read_link(release_root.join("current")).await.unwrap();
311 + assert_eq!(target.to_string_lossy(), "releases/0.2.0");
312 + let via_current = tokio::fs::read(release_root.join("current/server")).await.unwrap();
313 + assert_eq!(via_current, b"V2");
314 + }
315 +
316 + #[tokio::test]
317 + async fn gc_local_releases_keeps_last_n_by_mtime() {
318 + // Build > RELEASES_TO_KEEP fake release dirs with distinct mtimes,
319 + // then run gc and check which survived.
320 + let tmp = tempfile::tempdir().unwrap();
321 + let root = tmp.path();
322 + let releases = root.join("releases");
323 + tokio::fs::create_dir_all(&releases).await.unwrap();
324 +
325 + let total = RELEASES_TO_KEEP + 3;
326 + let mut names = Vec::new();
327 + for i in 0..total {
328 + let name = format!("v{i:02}");
329 + let dir = releases.join(&name);
330 + tokio::fs::create_dir(&dir).await.unwrap();
331 + // Stagger mtimes deterministically. tokio's File doesn't expose
332 + // set_times, so reach for std::fs::File + std::fs::FileTimes
333 + // (stable since 1.75). Synchronous is fine here — this is test
334 + // setup, not the hot path.
335 + let f = std::fs::File::open(&dir).unwrap();
336 + let when = SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(1_700_000_000 + i as u64);
337 + let times = std::fs::FileTimes::new().set_modified(when);
338 + f.set_times(times).unwrap();
339 + names.push(name);
340 + }
341 +
342 + gc_local_releases(root).await.unwrap();
343 +
344 + // The last RELEASES_TO_KEEP by mtime (i.e. highest i) survive.
345 + let surviving_expected: Vec<_> = names
346 + .iter()
347 + .skip(total - RELEASES_TO_KEEP)
348 + .cloned()
349 + .collect();
350 + for name in &surviving_expected {
351 + assert!(
352 + releases.join(name).exists(),
353 + "expected to survive: {name}"
354 + );
355 + }
356 + for name in names.iter().take(total - RELEASES_TO_KEEP) {
357 + assert!(
358 + !releases.join(name).exists(),
359 + "expected to be pruned: {name}"
360 + );
361 + }
362 + }
363 +
364 + #[tokio::test]
365 + async fn gc_local_releases_noop_when_below_threshold() {
366 + let tmp = tempfile::tempdir().unwrap();
367 + let root = tmp.path();
368 + let releases = root.join("releases");
369 + tokio::fs::create_dir_all(&releases).await.unwrap();
370 + for i in 0..3 {
371 + tokio::fs::create_dir(releases.join(format!("v{i}"))).await.unwrap();
372 + }
373 + gc_local_releases(root).await.unwrap();
374 + for i in 0..3 {
375 + assert!(releases.join(format!("v{i}")).exists());
376 + }
377 + }
378 +
379 + #[tokio::test]
380 + async fn gc_local_releases_noop_when_releases_dir_missing() {
381 + let tmp = tempfile::tempdir().unwrap();
382 + gc_local_releases(tmp.path()).await.unwrap();
383 + }
384 +
385 + #[tokio::test]
386 + async fn deploy_remote_fails_cleanly_when_host_unreachable() {
387 + // 192.0.2.0/24 is reserved for documentation and routes nowhere.
388 + // ConnectTimeout=10 limits the test wallclock to ~10s worst case.
389 + let tmp = tempfile::tempdir().unwrap();
390 + let staged = tmp.path().join("releases").join("0.0.1");
391 + tokio::fs::create_dir_all(&staged).await.unwrap();
392 + tokio::fs::write(staged.join("server"), b"x").await.unwrap();
393 +
394 + let node = crate::topology::Node {
395 + name: "unreachable".into(),
396 + ssh_target: "deploy@192.0.2.1".into(),
397 + release_root: "/opt/never".into(),
398 + service_name: "makenotwork.service".into(),
399 + };
400 +
401 + let result = deploy_node(&node, "0.0.1", &staged, "server").await;
402 + let err = result.expect_err("deploy to unreachable host should fail");
403 + let msg = format!("{err:#}");
404 + // The ssh helper returns `ssh <target> failed: ...`. Don't pin the
405 + // exact wording, just that the failure is attributed and that no
406 + // panic / hang happened.
407 + assert!(
408 + msg.contains("ssh") || msg.contains("rsync") || msg.contains("connection"),
409 + "unexpected error: {msg}"
410 + );
411 + }
412 +
413 + #[tokio::test]
414 + async fn deploy_node_with_local_ssh_target_swaps_symlink() {
415 + // ssh_target="local" should route to the local fast-path: just a
416 + // symlink swap, no remote calls. Helpful for dev loops.
417 + let tmp = tempfile::tempdir().unwrap();
418 + let release_root = tmp.path().to_path_buf();
419 + let staged = release_root.join("releases").join("0.0.1");
420 + tokio::fs::create_dir_all(&staged).await.unwrap();
421 + tokio::fs::write(staged.join("server"), b"x").await.unwrap();
422 +
423 + let node = crate::topology::Node {
424 + name: "local-dev".into(),
425 + ssh_target: "local".into(),
426 + release_root: release_root.to_string_lossy().into_owned(),
427 + service_name: "makenotwork.service".into(),
428 + };
429 +
430 + let out = deploy_node(&node, "0.0.1", &staged, "server").await.unwrap();
431 + assert_eq!(out, staged);
432 + let target = tokio::fs::read_link(release_root.join("current")).await.unwrap();
433 + assert_eq!(target.to_string_lossy(), "releases/0.0.1");
434 + }
71 435 }
M sando/daemon/src/gates.rs +122 -17
M sando/daemon/src/main.rs +9 -1
M sando/daemon/src/routes.rs +524 -23
M sando/daemon/src/state.rs +9
M sando/daemon/src/sync.rs +1
M sando/daemon/src/topology.rs +6
M sando/tui/src/main.rs +605 -93
A sando/daemon/src/events.rs +126
A sando/deploy/bootstrap-node.sh +164
A sando/deploy/sandod-backup-fetch.service +18
A sando/deploy/sandod-backup-fetch.timer +17
A sando/plans/config-artifacts.md +126
A sando/plans/migration-dryrun-failures.md +134