max / makenotwork
6 files changed,
+0 insertions,
-968 deletions
| @@ -1,126 +0,0 @@ | |||
| 1 | - | # Config artifacts vs binary artifacts | |
| 2 | - | ||
| 3 | - | Phase 3 design doc. Resolves: which of `deploy.sh`'s per-deploy actions sando absorbs, which move to one-time node-bootstrap, which sando explicitly skips. | |
| 4 | - | ||
| 5 | - | Status: draft. Decisions below are recommendations; checkboxes match `MNW/sando/todo.md` Phase 3. | |
| 6 | - | ||
| 7 | - | ## Inventory of `deploy.sh`'s actions | |
| 8 | - | ||
| 9 | - | | Action | Frequency | What it does | | |
| 10 | - | |------------------------------|----------------|----------------------------------------------------------------| | |
| 11 | - | | `build_binary` | per-deploy | cargo-zigbuild on macOS → x86_64 Linux musl/glibc | | |
| 12 | - | | `upload_config: Caddyfile` | per-deploy | scp `Caddyfile` → `/etc/caddy/Caddyfile`, `systemctl reload caddy` | | |
| 13 | - | | `upload_config: error-pages` | per-deploy | scp `error-pages/*.html` → `/opt/makenotwork/error-pages/` | | |
| 14 | - | | `upload_config: security` | per-deploy | scp `sshd-git.conf`, `fail2ban-sshd.conf`, `setup-firewall.sh` | | |
| 15 | - | | `upload_config: chmod` | per-deploy | chmod +x on setup-* scripts | | |
| 16 | - | | `upload_binary` | per-deploy | scp `makenotwork` + `mnw-admin` → `/opt/makenotwork/` | | |
| 17 | - | | `send_restart_warning` | per-deploy | POST `/api/internal/restart-warning` (30s notice), sleep 30s | | |
| 18 | - | | `restart_app` | per-deploy | `systemctl restart makenotwork`; curl 127.0.0.1:3000 to verify | | |
| 19 | - | | `sqlx migrate run` (implied) | startup | server runs migrations on startup in `main.rs:73` | | |
| 20 | - | ||
| 21 | - | ## Decision per item | |
| 22 | - | ||
| 23 | - | ### 1. Caddyfile — **bootstrap-only, not per-deploy** | |
| 24 | - | ||
| 25 | - | Caddy config is stable infrastructure. Most releases don't touch it. Per-deploy uploads couple binary version to config version unnecessarily and risk reload churn for unchanged config. | |
| 26 | - | ||
| 27 | - | - Node-bootstrap script installs `/etc/caddy/Caddyfile` once. | |
| 28 | - | - Updating Caddy config is an explicit operator action (`sando-cli push-caddy` or just `scp + systemctl reload caddy` manually), tracked but not per-release. | |
| 29 | - | - Revisit if Caddy config changes start landing >1x per sprint, then move to per-release artifact under `releases/<version>/Caddyfile` with a deploy hook. | |
| 30 | - | ||
| 31 | - | **Per-project alternative tracked:** if a Caddyfile change accompanies a binary change (rare), the operator must run the explicit Caddy-push step alongside `sando promote`. | |
| 32 | - | ||
| 33 | - | ### 2. error-pages — **bake into binary** | |
| 34 | - | ||
| 35 | - | Error pages version with code. They reference brand glyphs (diamond mark) and copy that drifts with the rest of the site. | |
| 36 | - | ||
| 37 | - | - Use `include_dir!` or `include_bytes!` to embed `server/deploy/error-pages/*.html` into the binary. | |
| 38 | - | - Update Caddy `handle_errors` blocks to point at an in-app fallback route (e.g. `/__errors/404.html`) instead of `/opt/makenotwork/error-pages/`. That route can serve the embedded HTML. | |
| 39 | - | ||
| 40 | - | Cost: small MNW server PR (separate from sando). Marks `deploy.sh upload_config: error-pages` step removable. | |
| 41 | - | ||
| 42 | - | Until that lands: ship error-pages as sibling under `releases/<version>/error-pages/`. Caddy still reads from `/opt/makenotwork/error-pages/` symlinked to `current/error-pages`. (Track A on testnot already has the `current` symlink working; just symlink error-pages parallel.) | |
| 43 | - | ||
| 44 | - | ### 3. mnw-admin binary — **ship alongside server** | |
| 45 | - | ||
| 46 | - | `mnw-admin` is part of the release; deploy.sh uploads it. Sando should too. | |
| 47 | - | ||
| 48 | - | - Extend `cfg.bin_name: String` → `cfg.bin_names: Vec<String>` (e.g. `["makenotwork", "mnw-admin"]`). | |
| 49 | - | - `deploy_local` + `deploy_node` iterate over the list, rsyncing each to `releases/<version>/<bin>`. | |
| 50 | - | - Build step looks up each in `server/target/release/<bin>`. | |
| 51 | - | ||
| 52 | - | Default stays `["server"]` for backwards-compat with the existing example config. | |
| 53 | - | ||
| 54 | - | ### 4. systemd unit (`makenotwork.service`) — **bootstrap-only** | |
| 55 | - | ||
| 56 | - | The unit references `<release_root>/current/makenotwork`. Once installed, it doesn't change per release. | |
| 57 | - | ||
| 58 | - | - Node-bootstrap script installs `/etc/systemd/system/makenotwork.service`. | |
| 59 | - | - `deploy.sh`'s upload of the unit was a re-upload-every-time pattern. Sando does not. | |
| 60 | - | - If the unit ever needs to change (e.g. resource limits, env file path), that's a one-shot operator action, not a per-deploy step. | |
| 61 | - | ||
| 62 | - | ### 5. Security configs (sshd-git, fail2ban, firewall) — **bootstrap-only** | |
| 63 | - | ||
| 64 | - | These are one-time host hardening. They have no release coupling. | |
| 65 | - | ||
| 66 | - | - Node-bootstrap script installs them on first provision. | |
| 67 | - | - Updates are out-of-band operator actions (or fold into a `sando push-config` later). | |
| 68 | - | ||
| 69 | - | ### 6. backup-db.sh — **bootstrap-only** | |
| 70 | - | ||
| 71 | - | Same as security configs. Backup script is host infrastructure, not release artifact. | |
| 72 | - | ||
| 73 | - | - Node-bootstrap installs `backup-db.sh` and its cron entry. | |
| 74 | - | - Updates out-of-band. | |
| 75 | - | - Bonus: backup-db.sh should be updated to (a) maintain `latest.sql.gz` hard link, (b) push to astra for true offsite — currently broken (see separate "offsite sync broken" ticket). | |
| 76 | - | ||
| 77 | - | ### 7. Restart warning — **defer to Phase 5; track for prod cutover** | |
| 78 | - | ||
| 79 | - | `deploy.sh` posts a 30s warning, sleeps 30s, then restarts. Sando does NOT yet do this. | |
| 80 | - | ||
| 81 | - | - For testnot (low traffic): skip. Service crash-loops invisibly enough. | |
| 82 | - | - For prod cutover: sando must implement this. Options: | |
| 83 | - | - **A**: Sando POSTs `/api/internal/restart-warning` itself, requires CLI_SERVICE_TOKEN exposed to sando. Token would live in `/etc/sando/sando.env` on fw13. | |
| 84 | - | - **B**: Sando exposes a `pre_deploy_hook` per-tier in `sando.toml` (shell command); operator decides. | |
| 85 | - | - Recommendation: **A** for prod tiers only (`tier.restart_warning_seconds = 30` in `sando.toml`). Tier A (testnot) leaves it unset = no warning. | |
| 86 | - | ||
| 87 | - | Phase 5 implementation, not blocking cutover-readiness. | |
| 88 | - | ||
| 89 | - | ### 8. Cross-compile from macOS — **retire** | |
| 90 | - | ||
| 91 | - | fw13 is x86_64 Ubuntu-derived, prod is x86_64 Ubuntu 24.04. Sando builds natively. Cargo-zigbuild path goes away once sando is canonical. | |
| 92 | - | ||
| 93 | - | - Verify: take a recent prod binary (from `deploy.sh`'s build) and sando's binary for the same sha, compare runtime behavior across one full sprint of testnot use. | |
| 94 | - | - Once verified, mark `deploy.sh` archived and delete cargo-zigbuild from dev-machine setup notes. | |
| 95 | - | ||
| 96 | - | ### 9. Prod migrations — **server-self-applies on startup; sando does NOT** | |
| 97 | - | ||
| 98 | - | MNW server runs `sqlx::migrate!("./migrations").run(&db).await` in `main.rs:73` at startup. This means: | |
| 99 | - | ||
| 100 | - | - A new binary starting up applies any pending migrations against the live prod DB. | |
| 101 | - | - Sando does not need an explicit `POST /migrate/{tier}` endpoint. | |
| 102 | - | - The `migration_dry_run` gate's purpose is to catch migration FAILURE before the live binary tries to run them — that's the prod safety net. | |
| 103 | - | - Risk: a partially-applied migration (e.g. multi-statement, the 2026-05-22 incident class) can leave the DB in a broken state mid-startup. Sandbox the migration via `migration_dry_run` catches this; the live server then either succeeds or fails-and-crash-loops on the same migration sequence. | |
| 104 | - | - Open question: should sando refuse to promote if `migration_dry_run` flags the upcoming version as a destructive migration (drop+recreate column)? Phase 5+ enhancement. | |
| 105 | - | ||
| 106 | - | **Action:** none — current architecture is correct. Document this in `plans/migration-dryrun-failures.md` (Phase 2 follow-up). | |
| 107 | - | ||
| 108 | - | ## Net effect on `deploy.sh` | |
| 109 | - | ||
| 110 | - | | Step | Replaced by Sando | Moved to node-bootstrap | Retired | | |
| 111 | - | |---------------------|------------------------------|-------------------------|---------| | |
| 112 | - | | build_binary | yes (native on fw13) | | | | |
| 113 | - | | upload_config | | yes (Caddyfile, etc.) | | | |
| 114 | - | | upload_binary | yes (+ mnw-admin) | | | | |
| 115 | - | | send_restart_warning| yes (Phase 5, prod tier only)| | | | |
| 116 | - | | restart_app | yes (reload-or-restart) | | | | |
| 117 | - | ||
| 118 | - | Once items 2-9 above land, `deploy.sh` becomes redundant and moves to `server/deploy/archive/`. | |
| 119 | - | ||
| 120 | - | ## Implementation order | |
| 121 | - | ||
| 122 | - | 1. **`bin_names: Vec<String>`** — small, unblocks mnw-admin shipping (#3). | |
| 123 | - | 2. **error-pages as release sibling + symlink** — small, unblocks #2 until bake-into-binary lands. | |
| 124 | - | 3. **node-bootstrap script** — folds Caddyfile (#1), unit (#4), security (#5), backup (#6) into one idempotent script. Already a Phase 1 carryover. | |
| 125 | - | 4. **Phase 5: restart_warning hook** — when prod cutover gets scheduled. | |
| 126 | - | 5. **Prod cutover sprint** — verify binary parity (#8), retire `deploy.sh` (#9 needs no action). |
| @@ -1,134 +0,0 @@ | |||
| 1 | - | # migration_dry_run: failure modes + what the operator sees | |
| 2 | - | ||
| 3 | - | `migration_dry_run` is the load-bearing pre-flight gate. It restores the | |
| 4 | - | latest prod backup into the scratch postgres, then runs the worktree's | |
| 5 | - | migrations on top. The point is to catch migrations that work on a fresh DB | |
| 6 | - | but break on real prod data, *before* the binary tries them in production. | |
| 7 | - | ||
| 8 | - | Gate definition (`MNW/sando/daemon/src/gates.rs::migration_dry_run`): | |
| 9 | - | ||
| 10 | - | 1. Look up latest row in `backups` table (set by `POST /backup/fetch`). | |
| 11 | - | 2. `reset_scratch` — drop every non-system schema, recreate `public`. | |
| 12 | - | 3. `restore_dump` — `gunzip -c <backup> | psql <scratch_url>`. | |
| 13 | - | 4. `run_migrator` — `sqlx::migrate::Migrator::new(<worktree>/server/migrations).run()`. | |
| 14 | - | ||
| 15 | - | All four steps' failure cases below. | |
| 16 | - | ||
| 17 | - | ## Operator surface | |
| 18 | - | ||
| 19 | - | Each invocation writes a row to `gate_runs`: | |
| 20 | - | ||
| 21 | - | | col | meaning | | |
| 22 | - | |--------------|---------------------------------------------------------------| | |
| 23 | - | | version | the version being built | | |
| 24 | - | | tier | always "mm" | | |
| 25 | - | | gate_kind | "migration_dry_run" | | |
| 26 | - | | passed | 1 = green, 0 = red | | |
| 27 | - | | detail | error tail (up to ~4 KB) — what to read first | | |
| 28 | - | | finished_at | wall-clock end | | |
| 29 | - | ||
| 30 | - | In the WS `/events` stream you also get `gate_start` and `gate_done` envelopes. | |
| 31 | - | The TUI surfaces these in the gate strip; on red, click into `detail`. | |
| 32 | - | ||
| 33 | - | ## Known failure modes | |
| 34 | - | ||
| 35 | - | ### 1. No backup fetched | |
| 36 | - | ||
| 37 | - | `detail = "no backup fetched; call /backup/fetch first"` | |
| 38 | - | ||
| 39 | - | Cause: `backups` table is empty (sando state DB is fresh, or the daily timer | |
| 40 | - | hasn't fired yet). Recovery: `curl -X POST $SANDO_DAEMON/backup/fetch`. | |
| 41 | - | ||
| 42 | - | ### 2. scratch_db_url unset | |
| 43 | - | ||
| 44 | - | `detail = "scratch_db_url unset in daemon config"` | |
| 45 | - | ||
| 46 | - | Cause: `/etc/sando/sando-daemon.toml` is missing the `scratch_db_url` line. | |
| 47 | - | Recovery: add it, restart sandod. | |
| 48 | - | ||
| 49 | - | ### 3. Scratch reset failed | |
| 50 | - | ||
| 51 | - | `detail = "scratch reset: <pg-error>"` | |
| 52 | - | ||
| 53 | - | Cause: postgres is down on the Sando host, or the role/db disappeared. | |
| 54 | - | Recovery: `systemctl status postgresql`, recreate `sando` role + `sando_scratch` | |
| 55 | - | db if needed (see bootstrap-node.sh template). | |
| 56 | - | ||
| 57 | - | ### 4. Restore failed | |
| 58 | - | ||
| 59 | - | `detail = "restore: <gunzip|psql error>"` | |
| 60 | - | ||
| 61 | - | Cause: backup file is corrupt or truncated. Verify with | |
| 62 | - | `zcat /srv/sando/backups/latest.sql.gz | head` — should start with `--` | |
| 63 | - | postgres dump preamble. If corrupt: re-fetch from prod with | |
| 64 | - | `POST /backup/fetch`. If repeatable, prod's backup itself is bad. | |
| 65 | - | ||
| 66 | - | ### 5. Migration drift (the load-bearing case) | |
| 67 | - | ||
| 68 | - | `detail = "migration <N> was previously applied but is missing in the resolved migrations"` | |
| 69 | - | ||
| 70 | - | **This is the gate doing its job.** The backup carries | |
| 71 | - | `_sqlx_migrations` rows for every migration prod has applied. If the worktree | |
| 72 | - | is missing one of those files, sqlx refuses to run — because applying the | |
| 73 | - | worktree against this DB would skip a step prod thinks is done. | |
| 74 | - | ||
| 75 | - | **Real example (2026-05-31):** sha `eee96a7` was pushed to sandod before | |
| 76 | - | migrations 123-132 landed in main. `migration_dry_run` failed with | |
| 77 | - | "migration 123 was previously applied but is missing in the resolved | |
| 78 | - | migrations". Recovery: push the up-to-date `main`. | |
| 79 | - | ||
| 80 | - | This is also the only signal you get for a forgotten migration file. If | |
| 81 | - | someone deletes `migrations/123_foo.sql` without truly rolling back the | |
| 82 | - | schema change in prod, this gate is what catches it. | |
| 83 | - | ||
| 84 | - | ### 6. Migration content changed | |
| 85 | - | ||
| 86 | - | `detail = "migration <N> was previously applied but has been modified"` | |
| 87 | - | (or similar — sqlx phrasing varies by version). | |
| 88 | - | ||
| 89 | - | Cause: the worktree's `123_foo.sql` has a different checksum than the version | |
| 90 | - | prod recorded in `_sqlx_migrations`. Don't fix by overwriting prod's | |
| 91 | - | checksum — that hides a real divergence. Investigate which version was | |
| 92 | - | "right" and add a follow-up migration that produces the intended state. | |
| 93 | - | ||
| 94 | - | ### 7. Migration content broken against prod data | |
| 95 | - | ||
| 96 | - | `detail = "migration <N>: <pg syntax/constraint/data error>"` | |
| 97 | - | ||
| 98 | - | Cause: the new migration runs fine on an empty schema (which is what | |
| 99 | - | local-dev `cargo test` exercises) but fails on actual prod data. Examples: | |
| 100 | - | adding `NOT NULL` to a column with existing nulls; `DROP COLUMN` referenced | |
| 101 | - | by a view; `UNIQUE` constraint violated by existing rows. | |
| 102 | - | ||
| 103 | - | **This is the 2026-05-22-incident class.** Without `migration_dry_run`, the | |
| 104 | - | binary deploys, starts up, runs the migration, partially-applies it, exits | |
| 105 | - | non-zero, systemd crash-loops, prod is down. With the gate, the failure | |
| 106 | - | happens on the Sando host's scratch DB and prod stays up. | |
| 107 | - | ||
| 108 | - | Recovery: fix the migration. Common patterns: | |
| 109 | - | - `NOT NULL` + default + then alter to non-default | |
| 110 | - | - `DROP COLUMN` only after dropping dependents | |
| 111 | - | - backfill via separate migration before the constraint | |
| 112 | - | ||
| 113 | - | ## Things this gate does NOT catch | |
| 114 | - | ||
| 115 | - | - Migrations that succeed but break the *application* — only caught by | |
| 116 | - | `cargo_test` (red) or `boot_smoke` (binary fails to start). | |
| 117 | - | - Migrations that are slow on prod-scale data (scratch DB has prod *content* | |
| 118 | - | but no prod *load*). | |
| 119 | - | - Migrations that need privileged operations not granted to `sando` role | |
| 120 | - | (the scratch role isn't superuser by design — see Phase 0 decisions). | |
| 121 | - | ||
| 122 | - | ## Operator playbook (red gate) | |
| 123 | - | ||
| 124 | - | 1. Read `detail`. It's almost always the right answer in the first line. | |
| 125 | - | 2. If it's a drift case (#5), check `git log -- server/migrations/` for what | |
| 126 | - | prod has that the worktree doesn't. | |
| 127 | - | 3. If it's a content failure (#7), reproduce locally: | |
| 128 | - | `cargo sqlx migrate run --source server/migrations` against a freshly | |
| 129 | - | restored backup. Iterate on the migration file. | |
| 130 | - | 4. Push the fix. Sandod's mutex aborts the old build; the new sha runs the | |
| 131 | - | gate from scratch. | |
| 132 | - | 5. Never bypass — there's no `?force=true` and there shouldn't be. If you | |
| 133 | - | really need to ship around a known-bad migration, that's an explicit | |
| 134 | - | `--hotfix` promote, and you own the prod consequences. |
| @@ -1,91 +0,0 @@ | |||
| 1 | - | # MakeMachine Hardware BOM | |
| 2 | - | ||
| 3 | - | Settled 2026-05-23. Top-of-line host platform; GPUs are fungible and live on the EveryCycle GPU thread (see `~/Code/everycycle/docs/roadmap.md`). | |
| 4 | - | ||
| 5 | - | The substrate is built once and kept stable; GPU experimentation happens above it without revisiting motherboard, CPU, or RAM. | |
| 6 | - | ||
| 7 | - | ## Component list | |
| 8 | - | ||
| 9 | - | | Component | Choice | Approx cost | Notes | | |
| 10 | - | |---|---|---|---| | |
| 11 | - | | CPU | AMD Threadripper Pro 7975WX | $3,900 | 32 cores, 5.3 GHz boost, 8-channel DDR5. Same memory controller and PCIe lanes as bigger SKUs; cores past ~32 starve on memory bandwidth for inference. | | |
| 12 | - | | Motherboard | ASRock Rack WRX90D8-2L/2T | $1,200 | 7× PCIe 5.0 ×16, 8 DIMM slots, **ASPEED AST2600 BMC (OpenBMC-friendly)**, dual 10 GbE, 4× M.2 + 4× SlimSAS. | | |
| 13 | - | | RAM | 8× 64 GB DDR5-5600 ECC RDIMM (512 GB total) | $2,500 | One DIMM per channel — 2 DIMMs per channel forces DDR5 down to ~4400 MT/s on WRX90, costing ~20% memory bandwidth (real impact on CPU-offload layers). Brand: Micron or Hynix off the board QVL. | | |
| 14 | - | | Storage 1 | 4 TB Gen5 NVMe (Samsung 9100 Pro or Crucial T705) | $600 | ZFS root pool: OS, models, Sando state, logs. | | |
| 15 | - | | Storage 2 | 4 TB Gen5 NVMe (same) | $600 | **Raw XFS kv-scratch.** Not ZFS — ZFS caps Gen5 throughput; scratch is by definition disposable. EveryCycle uses this for kv-cache overflow. | | |
| 16 | - | | PSU | 2000 W Titanium-class (Super Flower Leadex Titanium or equivalent) | $600 | Headroom for any GPU combination the loose-parts experimentation hits. | | |
| 17 | - | | Chassis | 4U rackmount with ≥4 dual-slot GPU bays (Sliger CX4712 or similar) | $400 | Matches the eventual EveryCycle reference inference box; rackable from day one. | | |
| 18 | - | | CPU cooler | Silverstone XE360-TR5 (air) or Noctua NH-U14S TR5-SP6 | $200 | Air, not AIO — pump failure on a 24/7 box is worse than fan failure. | | |
| 19 | - | | Case fans | 6× 140 mm Noctua industrial | $200 | Front-to-back airflow; GPUs in a 4U breathe through these. | | |
| 20 | - | | Boot GPU | Nvidia GT 1030 (low-profile) | $80 | WRX90 has no iGPU; need a tiny card for console/boot. Also used as console GPU when datacenter cards (no display output) are installed. GT 710 originally specced but effectively EOL retail in 2026; GT 1030 is the current floor. Alternative: skip entirely if WRX90 BMC serial-over-LAN proves reliable. | | |
| 21 | - | | **Host subtotal** | | **~$10,280** | Before any compute GPU. | | |
| 22 | - | ||
| 23 | - | ## First compute GPU (Thread A1) | |
| 24 | - | ||
| 25 | - | | Component | Choice | Approx cost | Notes | | |
| 26 | - | |---|---|---|---| | |
| 27 | - | | GPU | 1× Nvidia Tesla P40 | $200 | 24 GB GDDR5, Pascal (cc 6.1). On-thesis: Pascal is the next architecture facing CUDA-legacy transition. | | |
| 28 | - | | Cooling adapter | 3D-printed or commercial fan shroud for P40 | $25 | Server card; passive cooling needs chassis airflow OR a strapped-on fan. | | |
| 29 | - | | Power adapter | EPS 8-pin to PCIe (or proper EPS routing) | $10 | P40 uses CPU-style 8-pin, not standard PCIe 8-pin. | | |
| 30 | - | | **GPU subtotal at A1** | | **~$235** | | | |
| 31 | - | ||
| 32 | - | ## Total | |
| 33 | - | ||
| 34 | - | | | | | |
| 35 | - | |---|---| | |
| 36 | - | | Host platform | ~$10,280 | | |
| 37 | - | | First GPU (A1) | ~$235 | | |
| 38 | - | | **MakeMachine v0 (assembled, runnable)** | **~$10,515** | | |
| 39 | - | ||
| 40 | - | Budget originally specced at $14–16K for the previous spec. Net savings: ~$4–5K. The savings are the GPU experimentation budget for advancing the GPU thread through A4–A5 over time. | |
| 41 | - | ||
| 42 | - | ## BIOS settings worth confirming at first boot | |
| 43 | - | ||
| 44 | - | - **Above 4G Decoding: enabled.** Required for datacenter cards with large BARs (Tesla P40, MI50, etc.). | |
| 45 | - | - **Resizable BAR: enabled.** Same reason. | |
| 46 | - | - **IOMMU: enabled.** Required for podman + CDI GPU passthrough. | |
| 47 | - | - **SR-IOV: as needed.** Not critical at A1; revisit at A5. | |
| 48 | - | - **Memory speed: DDR5-5600 (JEDEC).** Not pushed past spec. | |
| 49 | - | - **PCIe lane bifurcation: leave default initially; revisit for multi-GPU configurations.** | |
| 50 | - | ||
| 51 | - | ## Why this spec, summarized | |
| 52 | - | ||
| 53 | - | - **Threadripper Pro WRX90 platform** chosen over consumer TRX50 for the 7× PCIe 5.0 ×16 slots and the AST2600 BMC. EveryCycle's `bmc-agent` crate needs a real BMC to talk to. | |
| 54 | - | - **512 GB RAM** lets a 405B Q4 model fit fully in RAM if ever wanted; the immediate need is comfortable headroom for kv-cache experiments. | |
| 55 | - | - **Two Gen5 NVMes with different filesystems** because ZFS root is the right answer for the OS and model store, but ZFS would cap Gen5 bandwidth on the kv-scratch where bandwidth is the whole point. | |
| 56 | - | - **2000 W PSU** is sized for the worst-case 4-GPU configuration on the GPU thread, not for A1 alone. | |
| 57 | - | - **4U rackmount chassis** because it doubles as a prototype of the eventual EveryCycle reference appliance form factor. | |
| 58 | - | ||
| 59 | - | ## Sourcing notes | |
| 60 | - | ||
| 61 | - | Most of the spine is specialist channels; Microcenter covers the commodity parts. | |
| 62 | - | ||
| 63 | - | | Part | Channel | | |
| 64 | - | |---|---| | |
| 65 | - | | TR Pro 7975WX | Newegg, ShopBLT, Provantage | | |
| 66 | - | | WRX90D8-2L/2T | ASRock Rack direct, ShopBLT (longest lead — order first) | | |
| 67 | - | | DDR5-5600 ECC RDIMM (Micron/Hynix off board QVL) | Nemix, ServerSupply, Newegg | | |
| 68 | - | | Sliger CX4712 | Sliger direct (~6 wk historical lead). Fallback: Rosewill RSV-L4500U from Newegg if waiting is unacceptable. | | |
| 69 | - | | Tesla P40 + EPS-to-PCIe adapter + fan shroud | eBay | | |
| 70 | - | | 2× 4 TB Gen5 NVMe (9100 Pro / T705) | Microcenter | | |
| 71 | - | | PSU | Microcenter if Super Flower Leadex Titanium 2000 W in stock; otherwise Seasonic PRIME TX-1600 is the MC-reliable alternative (sufficient for A1–A3, marginal at A4 4-GPU). | | |
| 72 | - | | Noctua NF-A14 industrialPPC-3000 fans | Microcenter | | |
| 73 | - | | NH-U14S TR5-SP6 cooler | Noctua direct or Newegg; Microcenter rarely stocks | | |
| 74 | - | | GT 1030 boot GPU, paste, cables, M.2 heatsinks | Microcenter | | |
| 75 | - | ||
| 76 | - | Suggested order of operations: board first (longest lead) → CPU + RAM kit together (QVL match) → chassis → Microcenter run for commodity parts → P40 last (cheapest, most fungible). | |
| 77 | - | ||
| 78 | - | ## Open considerations (not blocking purchase) | |
| 79 | - | ||
| 80 | - | - **PSU redundancy.** Single 2000 W Titanium is fine for v0. Dual-redundant CRPS (e.g. FSP Twins) is the server-class move for v1. | |
| 81 | - | - **Cooler headroom.** NH-U14S TR5-SP6 is adequate for the 7975WX's 350 W TDP, not generous. XE360-TR5 (listed alt) gives more headroom for sustained all-core + GPU host load. Both are air. | |
| 82 | - | - **Boot GPU skip.** If WRX90 BMC serial-over-LAN proves reliable, GT 1030 line item is removable and a slot is reclaimed. | |
| 83 | - | - **kv-scratch PLP.** Consumer 9100 Pro / T705 have no power-loss protection. Fine for disposable scratch; revisit if EveryCycle ever checkpoints kv-cache through it. | |
| 84 | - | ||
| 85 | - | ## What this BOM deliberately does not include | |
| 86 | - | ||
| 87 | - | - **Multiple compute GPUs at purchase time.** Each GPU thread milestone is a discrete add; advancing the thread is its own decision moment. | |
| 88 | - | - **Liquid cooling.** More maintenance overhead than a single-operator shop should carry. | |
| 89 | - | - **Optane / SLC drives.** Optane is discontinued; Gen5 TLC is the right answer now. | |
| 90 | - | - **128-core CPU.** Memory-bandwidth-starved for inference; ~$6K extra for no inference gain. | |
| 91 | - | - **A second NIC card.** The 2× 10 GbE onboard is enough for v0. |
| @@ -1,213 +0,0 @@ | |||
| 1 | - | # Sando observability architecture | |
| 2 | - | ||
| 3 | - | Status: draft, 2026-06-03. Argument shape, not a checklist. | |
| 4 | - | ||
| 5 | - | Goal: an error and visibility surface built on newtypes. Sando's current daemon and TUI agree by string convention — `tier: String`, `version: String`, `gate: String`, `detail: Option<String>` — and every consumer (schema, WS payloads, TUI render) reparses or pattern-matches those strings independently. The result is that gate failure classification only exists as prose in `gate_runs.detail`, and the TUI's only recourse on red is "open `/logs/<version>/<gate>` and read the tail." This document proposes the type graph that replaces the strings, the boundary parses that mint the types, and the persistence + wire shapes that follow. | |
| 6 | - | ||
| 7 | - | This is one cohesive design. The launchplan splits it into "Phase B (typed GateOutcome)" and "Phase C (live tail + remaining newtypes)"; that split is a delivery convenience, not an architectural seam. The migration order below preserves the option to ship Phase B first, but the types it introduces must already be the final shape, not a string stand-in to be newtyped later. | |
| 8 | - | ||
| 9 | - | ## What is currently stringly-typed | |
| 10 | - | ||
| 11 | - | Inventory from a read of `daemon/src/gates.rs`, `daemon/src/events.rs`, `daemon/src/topology.rs`, `daemon/migrations/001_init.sql`, and `tui/src/main.rs:186-266`: | |
| 12 | - | ||
| 13 | - | | Concept | Current shape | Where it appears | | |
| 14 | - | |---|---|---| | |
| 15 | - | | Tier | `tier: String` | `GateCtx`, every `Event` variant, `tiers.name`, `gate_runs.tier`, `tier_state.tier`, `deploys.tier`, TUI `str_v(v.get("tier"))` | | |
| 16 | - | | Node | `node: String` (sometimes `Option`) | `Tier.nodes[].name`, `deploys.node`, `Event::Deploy*` | | |
| 17 | - | | Version | `version: String` (semver `0.8.12`) | `GateCtx`, `versions.version`, every event, TUI display | | |
| 18 | - | | Git sha | `sha: String` (short or full?) | `versions.git_sha`, `Event::RebuildRequested/BuildStart/BuildOk/BuildFailed/BuildAborted`, `/logs/<sha>/<gate>` per launchplan §3 | | |
| 19 | - | | Gate kind | `Gate` enum *inside the daemon*; `gate_kind: &'static str` once it crosses the events/schema boundary; reparsed in TUI | `gates::kind_str`, `gate_runs.gate_kind`, `Event::Gate*.gate`, TUI `match kind` | | |
| 20 | - | | Gate outcome detail | `detail: Option<String>` | `GateOutcome.detail`, `gate_runs.detail`, conflates: config error, missing prerequisite, real failure with stderr tail, burn-in clock progress, "waiting on operator" | | |
| 21 | - | | Deploy outcome | `outcome TEXT DEFAULT 'in_progress'` | `deploys.outcome`; no enum on the Rust side | | |
| 22 | - | | Deploy failure | `error: String` | `Event::DeployFailed.error`; freeform | | |
| 23 | - | | Backup source | `source: String` | `backups.source`, `Event::BackupFetched.source`; an ssh URL today | | |
| 24 | - | | Canary policy | typed (`CanaryPolicy`) at config load, then `as_str()` to a `TEXT` column | `tiers.canary` | | |
| 25 | - | ||
| 26 | - | `gates::kind_str` and the TUI's `match kind` are the clearest tell — we have an enum on each side of the wire and a string in the middle. That string is the only schema, so adding a gate kind is a four-place edit with no compile-time enforcement. | |
| 27 | - | ||
| 28 | - | The same anti-pattern with higher stakes is `detail`. Three concrete examples from `gates.rs`: | |
| 29 | - | ||
| 30 | - | - `migration_dry_run` writes `"scratch_db_url unset in daemon config"` (a config bug, no point retrying). | |
| 31 | - | - `migration_dry_run` writes `"no backup fetched; call /backup/fetch first"` (a missing prerequisite, retry after `/backup/fetch`). | |
| 32 | - | - `boot_smoke` writes `"binary exited early: exit status: 101\n==== stdout ====\n…"` (a real failure carrying a tail). | |
| 33 | - | - `burn_in` writes `"47 hours remaining of 168"` (not a failure at all — the gate is correctly red but the deploy is not blocked by a defect). | |
| 34 | - | ||
| 35 | - | Today, an operator (or the TUI, or a future alerting rule) cannot tell those apart without substring-matching the prose. The point of the redesign is to make that classification a property of the value, not a property of how it's spelled. | |
| 36 | - | ||
| 37 | - | ## The type graph | |
| 38 | - | ||
| 39 | - | Domain types live in a new `daemon/src/domain.rs` (name negotiable). They are the vocabulary every other module speaks; they implement `Serialize`/`Deserialize`, `Display`, `FromStr`, and `sqlx::Type` so they round-trip through events, JSON responses, and SQLite columns without per-site conversion. | |
| 40 | - | ||
| 41 | - | ```text | |
| 42 | - | ┌──────────────┐ | |
| 43 | - | │ TierId │ newtype String, validated against topology on construction | |
| 44 | - | └──────────────┘ | |
| 45 | - | ┌──────────────┐ | |
| 46 | - | │ NodeId │ newtype String, validated against TierId's nodes | |
| 47 | - | └──────────────┘ | |
| 48 | - | ┌──────────────┐ | |
| 49 | - | │ Version │ parsed semver (semver::Version), Display = "0.8.12" | |
| 50 | - | └──────────────┘ | |
| 51 | - | ┌──────────────┐ | |
| 52 | - | │ GitSha │ enforced 40-hex or short-7-hex; one canonical form | |
| 53 | - | └──────────────┘ | |
| 54 | - | ┌──────────────┐ | |
| 55 | - | │ GateKind │ enum, replaces both `Gate` discriminant and `gate_kind` string | |
| 56 | - | │ │ Variants: CargoTest, MigrationDryRun, BootSmoke, | |
| 57 | - | │ │ BurnIn, ManualConfirm. Display = snake_case. | |
| 58 | - | └──────────────┘ | |
| 59 | - | ┌──────────────┐ | |
| 60 | - | │ GateRunId │ newtype i64, identifies one row of gate_runs | |
| 61 | - | └──────────────┘ | |
| 62 | - | ┌──────────────┐ | |
| 63 | - | │ DeployId │ newtype i64 | |
| 64 | - | └──────────────┘ | |
| 65 | - | ``` | |
| 66 | - | ||
| 67 | - | `Gate` (the topology struct with `BurnIn { hours }`) and `GateKind` (the discriminant) become distinct: `Gate` is the *config* (kind + parameters), `GateKind` is the *identifier* you use to talk about a class of gate in events and the schema. `gate_runs.gate_kind` stores `GateKind`; gate parameters at the time of the run, if we ever need them in history, become a separate column. | |
| 68 | - | ||
| 69 | - | ### `GateOutcome` redesign | |
| 70 | - | ||
| 71 | - | ```rust | |
| 72 | - | pub struct GateOutcome { | |
| 73 | - | pub status: GateStatus, | |
| 74 | - | pub log_ref: Option<LogRef>, // pointer to persisted stdout/stderr, not the tail itself | |
| 75 | - | } | |
| 76 | - | ||
| 77 | - | pub enum GateStatus { | |
| 78 | - | Passed { note: PassNote }, | |
| 79 | - | Failed(GateFailure), | |
| 80 | - | Blocked(GateBlocker), // gate cannot run yet; not a defect | |
| 81 | - | } | |
| 82 | - | ||
| 83 | - | pub enum PassNote { | |
| 84 | - | StayedUp { duration_s: u32 }, // boot_smoke | |
| 85 | - | BurnInElapsed { hours: u32 }, // burn_in | |
| 86 | - | Migrated { backup: BackupId, count: u32 }, // migration_dry_run | |
| 87 | - | TestsPassed { count: u32, duration_s: u32 }, // cargo_test | |
| 88 | - | OperatorConfirmed { at: DateTime<Utc> }, // manual_confirm | |
| 89 | - | } | |
| 90 | - | ||
| 91 | - | pub enum GateBlocker { | |
| 92 | - | BurnInClockNotStarted, | |
| 93 | - | BurnInRemaining { hours: u32, total: u32 }, | |
| 94 | - | AwaitingOperatorConfirmation, | |
| 95 | - | NoBackupAvailable, | |
| 96 | - | ScratchDbUrlUnset, | |
| 97 | - | ArtifactMissing { version: Version }, | |
| 98 | - | } | |
| 99 | - | ||
| 100 | - | pub enum GateFailure { | |
| 101 | - | CargoTest { failed_count: u32, first_failed: Option<String> }, | |
| 102 | - | MigrationDrift { migration: String }, // "previously applied but missing" | |
| 103 | - | MigrationModified { migration: String }, // "previously applied but modified" | |
| 104 | - | MigrationSqlError { migration: String, sqlstate: Option<String> }, | |
| 105 | - | RestoreFailed { kind: RestoreFailureKind }, | |
| 106 | - | BootPanic { exit_code: Option<i32> }, | |
| 107 | - | BootExitedEarly { exit_code: Option<i32> }, | |
| 108 | - | SpawnFailed { os_error: i32 }, | |
| 109 | - | Timeout { gate: GateKind, after_s: u32 }, | |
| 110 | - | Unclassified, // fallthrough; log_ref required | |
| 111 | - | } | |
| 112 | - | ``` | |
| 113 | - | ||
| 114 | - | Three things to notice about this shape: | |
| 115 | - | ||
| 116 | - | 1. **`Blocked` is its own variant.** Burn-in not yet elapsed and "scratch_db_url unset" are not failures — they are pre-conditions the operator can address out-of-band. The TUI can render them yellow instead of red. Today they're red, indistinguishably. | |
| 117 | - | 2. **`log_ref`, not `detail`.** The structured variants carry just enough to render a one-line summary (`migration 0047 modified`, `tests failed: 3`). The full tail lives on disk and the variant carries a pointer to it. This is the architectural seam between Phase B (classification) and Phase C (live tail) — once `log_ref` exists, Phase C is "ship chunks of that log over WS as they're written" rather than a redesign. | |
| 118 | - | 3. **`Unclassified` is admitted.** Classifiers are best-effort and the migration plan must work when a new failure mode shows up that no classifier matches yet. Unclassified failures still have a `log_ref`; they degrade gracefully to the current "read the tail" experience without breaking the contract. | |
| 119 | - | ||
| 120 | - | ### Classifier layer | |
| 121 | - | ||
| 122 | - | Each gate's runner produces raw output (`stdout`, `stderr`, exit status, plus any structured signals like a sqlx error). A classifier maps `(GateKind, RawOutput) -> GateStatus`. Classifiers are pure functions and live in `daemon/src/classify/` — one file per gate. The taxonomy in `plans/migration-dryrun-failures.md` is already 80% of `migration_dry_run`'s classifier; it gets ported into code, not just docs. | |
| 123 | - | ||
| 124 | - | Classifiers can be unit-tested with captured fixtures (`tests/fixtures/cargo_test_failed_compile.txt` → `GateStatus::Failed(GateFailure::CargoTest { failed_count: 0, .. })`). This is the first place Sando gets meaningful test coverage for diagnostic behavior, which `todo.md` flags as a gap. | |
| 125 | - | ||
| 126 | - | ### Where each type enters Sando | |
| 127 | - | ||
| 128 | - | Boundary discipline: every newtype is constructed once, at the edge where its string form enters the process. Internally, only the typed form moves. | |
| 129 | - | ||
| 130 | - | - `TierId`, `NodeId` — at `Topology::load`. The validator already walks tiers; it now mints `TierId`s and rejects unknown references at parse time instead of letting them surface as foreign-key failures. | |
| 131 | - | - `Version` — at the build step (`build.rs`), parsed from the server `Cargo.toml`. Stored as text in `versions.version`; the column round-trips through `sqlx::Type for Version`. | |
| 132 | - | - `GitSha` — at the post-receive hook entry point in `routes.rs::rebuild`. The hook today passes a string; the route normalizes to `GitSha` immediately. The `/logs/<sha>/<gate>` route (launchplan §3 wording) becomes `/logs/<GitSha>/<GateKind>`, but on-disk storage continues to key by `Version` until the build step has both available together. The comment at `gates.rs:432-434` already anticipates this transition. | |
| 133 | - | - `GateKind` — at `Topology` deserialization. `Gate` keeps its parameter-carrying variants; `GateKind` is derived from `Gate` and used everywhere else. | |
| 134 | - | - `GateOutcome` — produced by gate runners, persisted, emitted in events, displayed in TUI. Never lowered to a `String` along the way. | |
| 135 | - | ||
| 136 | - | ## Persistence | |
| 137 | - | ||
| 138 | - | `gate_runs` becomes: | |
| 139 | - | ||
| 140 | - | ```sql | |
| 141 | - | CREATE TABLE gate_runs ( | |
| 142 | - | id INTEGER PRIMARY KEY AUTOINCREMENT, | |
| 143 | - | version TEXT NOT NULL REFERENCES versions(version), | |
| 144 | - | tier TEXT NOT NULL REFERENCES tiers(name), | |
| 145 | - | gate_kind TEXT NOT NULL, -- GateKind, Display form | |
| 146 | - | started_at TEXT NOT NULL, | |
| 147 | - | finished_at TEXT, | |
| 148 | - | status TEXT, -- 'passed' | 'failed' | 'blocked' | NULL while in-flight | |
| 149 | - | outcome_json TEXT, -- serialized GateOutcome (PassNote / GateFailure / GateBlocker) | |
| 150 | - | log_ref TEXT -- relative path under cfg.logs_root, or NULL | |
| 151 | - | ); | |
| 152 | - | ``` | |
| 153 | - | ||
| 154 | - | `status` is denormalized for cheap indexing and `WHERE status = 'failed'` queries. `outcome_json` is the source of truth and is what the daemon reads back when serving `/state`. The migration drops `passed INTEGER` and `detail TEXT`, with a backfill that maps: | |
| 155 | - | ||
| 156 | - | - `passed = 1` → `status = 'passed'`, `outcome_json = {"kind":"passed","note":{"kind":"legacy","text":<old detail>}}` | |
| 157 | - | - `passed = 0`, detail matches a known prefix (`"burn-in"`, `"scratch_db_url unset"`, `"no backup fetched"`, `"waiting on operator"`) → `status = 'blocked'`, appropriate `GateBlocker` variant | |
| 158 | - | - `passed = 0` otherwise → `status = 'failed'`, `outcome_json = {"kind":"failed","failure":{"kind":"unclassified","legacy_detail":<old>}}` | |
| 159 | - | ||
| 160 | - | Backfill correctness is testable against the existing prod sandod sqlite — there are not many rows. | |
| 161 | - | ||
| 162 | - | `deploys.outcome` gets the same treatment in a smaller, separate migration: enum-typed (`InProgress | Ok | Failed { kind: DeployFailureKind }`), with the failure kind classifying spawn/transport/health-check distinctly from the freeform `error: String` carried by `Event::DeployFailed` today. | |
| 163 | - | ||
| 164 | - | ## Wire shape (events + `/state`) | |
| 165 | - | ||
| 166 | - | `Event::GateDone` becomes: | |
| 167 | - | ||
| 168 | - | ```rust | |
| 169 | - | GateDone { | |
| 170 | - | tier: TierId, | |
| 171 | - | version: Version, | |
| 172 | - | gate: GateKind, | |
| 173 | - | outcome: GateOutcome, | |
| 174 | - | } | |
| 175 | - | ``` | |
| 176 | - | ||
| 177 | - | The old `passed: bool` is gone — `outcome.status` carries strictly more information. The TUI's `format_event` becomes a match on the typed envelope (deserialized via `serde_json::from_str::<EventEnvelope>`, not `Value` reflection), and the per-kind render functions in `tui/src/main.rs:186-266` collapse into `Display` impls on the domain types. The `str_v` / `num_v` helpers stop being needed. | |
| 178 | - | ||
| 179 | - | `/state` (the TUI's polling endpoint) gains a `gates: Vec<GateRunSummary>` per tier, where `GateRunSummary` is the typed outcome plus timestamps and a `log_ref`. The TUI no longer needs `/logs/...` to populate its primary view; `/logs` becomes the drill-down for unclassified failures only. | |
| 180 | - | ||
| 181 | - | Phase C's live tail rides on the same `log_ref`: while a gate is in flight, `log_ref` is present and the WS emits `GateLogChunk { run_id: GateRunId, bytes: Vec<u8>, seq: u32 }` events as the runner flushes to disk. The TUI keeps a per-run ring buffer keyed by `GateRunId`. Because `GateRunId` is a newtype with a clear identity, the wire protocol does not need to invent a separate stream identifier. | |
| 182 | - | ||
| 183 | - | ## Migration order | |
| 184 | - | ||
| 185 | - | The order is constrained by what compiles together, not by the launchplan's B-then-C framing. | |
| 186 | - | ||
| 187 | - | 1. **Domain types module** — introduce `TierId`, `NodeId`, `Version`, `GitSha`, `GateKind`, `GateRunId`, `DeployId` with `Serialize` / `sqlx::Type` impls. No call sites change yet. Pure addition. | |
| 188 | - | 2. **Topology + config use the types** — `Topology::load` mints `TierId` / `NodeId`; `GateCtx` carries `TierId`, `Version`, not strings. `gate_runs.tier` and `gate_runs.version` columns stay `TEXT` (no schema migration); the change is on the Rust side only. `kind_str` deletes. | |
| 189 | - | 3. **Events use the types** — every `Event` variant takes the newtypes. WS frames are unchanged on the wire (snake-case strings) because `Serialize` impls produce identical JSON. The TUI keeps its `Value` reflection during this step; nothing forces it to break. | |
| 190 | - | 4. **`GateOutcome` redesign + classifier layer + `gate_runs` migration** — the biggest single step. Runners produce typed outcomes; classifiers live in `daemon/src/classify/`; schema migration 003 lands with backfill. `/state` exposes typed outcomes. TUI keeps `Value` reflection but is now reading typed fields (`status`, `outcome_json`). | |
| 191 | - | 5. **TUI typed event handling** — deserialize `EventEnvelope` directly, drop `str_v` / `num_v`, render via `Display` on domain types. Yellow for `Blocked`, red for `Failed`, green for `Passed`. | |
| 192 | - | 6. **Live tail (`GateLogChunk` events, `log_ref` plumbing)** — gates write to a per-run log file as they run, not at completion; runners drop the `Vec<u8>` buffers in `boot_smoke` / `cargo_test` in favor of streaming through a `tokio::fs::File` writer that also broadcasts chunks. TUI consumes. | |
| 193 | - | 7. **Deploy outcome typing + `DeployFailed` failure-kind classifier** — same pattern, smaller scope. Closes out the rest of the string surface. | |
| 194 | - | ||
| 195 | - | Steps 1–3 can ship in one commit per step without changing observable behavior. Step 4 is where the operator visibility actually changes; it's the smallest unit that justifies a version bump and a sando deploy to itself. | |
| 196 | - | ||
| 197 | - | ## Non-goals and open questions | |
| 198 | - | ||
| 199 | - | - **Not aiming for a generic gate framework.** The `GateFailure` variants are MNW-specific (cargo, sqlx migrations, `makenotwork` binary boot). If we ever gate non-MNW projects, this enum will need to factor; until then, concrete variants are clearer than `Box<dyn>` traits. | |
| 200 | - | - **Not introducing `Result<GateOutcome, GateError>`.** Runner-internal failures (sqlite I/O, can't spawn) collapse into `GateStatus::Failed(GateFailure::SpawnFailed | …)` or `GateBlocker` as appropriate. The outer `Result` only exists for genuine "the daemon itself is broken" cases that prevent persistence at all. | |
| 201 | - | - **Open: backfill aggressiveness.** Should we run classifiers retroactively against the historical `detail` strings during the schema-3 backfill, or only forward? Forward-only is simpler and the history isn't operator-load-bearing — leaning toward forward-only with the `unclassified + legacy_detail` envelope above, but worth deciding. | |
| 202 | - | - **Open: where does `GateRunId` originate?** Today the `INSERT … RETURNING id` returns a raw i64 inside `gates::run`. Cleanest is for `gates::run` to mint a `GateRunId` and pass it back; the event bus then carries it from `GateStart` through `GateLogChunk` through `GateDone`, making client-side correlation trivial. Worth confirming this doesn't break the `manual_confirm` lookup pattern in `gates.rs:386-403`. | |
| 203 | - | - **Open: `serde(tag = "kind")` vs explicit variant shape on `GateOutcome`.** The events module already uses `#[serde(tag = "kind", rename_all = "snake_case")]`, which would give us `{"kind":"failed","failure":{"kind":"migration_drift","migration":"0047"}}`. Two layers of tagged unions is verbose but consistent. Alternative: flatten via `#[serde(flatten)]` and accept that the JSON shape diverges slightly from the Rust shape. Leaning toward the explicit two-layer form because the TUI's parser then mirrors the Rust enum 1:1. | |
| 204 | - | - **Open: do we keep `detail TEXT` for one release as a shadow column, or drop in the same migration?** Shadow is safer for rollback. Adds clutter. Probably worth one release of shadow. | |
| 205 | - | ||
| 206 | - | ## Acceptance, when this is done | |
| 207 | - | ||
| 208 | - | - `gates.rs` contains no `String` literals describing failure modes. Every failure path constructs a `GateFailure` variant. | |
| 209 | - | - The TUI's `format_event` is gone; rendering goes through `Display` on domain types. | |
| 210 | - | - An operator hitting `c /api/state` sees structured outcomes (not just `"passed": false, "detail": "..."`). | |
| 211 | - | - A new gate kind is added by extending `GateKind`, `Gate`, and a classifier; the compiler enforces every other site. | |
| 212 | - | - `/logs/<version>/<gate>` is a fallback for `Unclassified` outcomes, not the primary diagnostic path. | |
| 213 | - | - Live tail works for `boot_smoke` and `cargo_test`; the buffered `Vec<u8>` pattern in `gates.rs:317-332` is gone. |
| @@ -1,241 +0,0 @@ | |||
| 1 | - | # Session 1 — Sando ships the full versioned bundle | |
| 2 | - | ||
| 3 | - | Plan captured 2026-06-02 after the design-step-back conversation following Phase A landing and the cargo_test/MM diagnosis push. Resolves the tier-B-strategy decision (§6.5 step 6 of `launchplan_final.md`) by going past the (a1)/(a2)/(b) trichotomy to a proper layout redesign. | |
| 4 | - | ||
| 5 | - | Status: **complete 2026-06-02**. See §G "Outcomes" at the bottom for what actually shipped, the bundle that landed, and what changed vs. the plan. | |
| 6 | - | ||
| 7 | - | ## Background — the trade as it stood | |
| 8 | - | ||
| 9 | - | Today sando ships only `{makenotwork, mnw-admin, error-pages}` into `releases/<v>/`. Prod's `/opt/makenotwork/` also contains `docs/`, `static/`, `yara-rules/`, `.env`, `backups/`, `scan-spool/` — none of which sando manages. `deploy.sh` ships content; sando ships binaries. The split is the source of the friction we keep hitting. | |
| 10 | - | ||
| 11 | - | The right answer isn't (a1) "extend sando to ship everything in one piece" or (a2) "leave content on the old path forever" — it's a layout that separates the five things prod actually serves and lets sando own exactly the right subset. | |
| 12 | - | ||
| 13 | - | ## Layout | |
| 14 | - | ||
| 15 | - | ``` | |
| 16 | - | /opt/mnw/releases/<v>/ # sando-managed: code + version-coupled content | |
| 17 | - | ├── makenotwork | |
| 18 | - | ├── mnw-admin | |
| 19 | - | ├── static/ | |
| 20 | - | ├── docs/ | |
| 21 | - | └── error-pages/ | |
| 22 | - | /opt/mnw/current → releases/<v>/ # atomic-swap symlink | |
| 23 | - | /opt/mnw/yara-rules/ # operator-managed (separate cadence) | |
| 24 | - | /etc/mnw/makenotwork.env # operator-managed secrets | |
| 25 | - | /var/lib/mnw/ # runtime state | |
| 26 | - | ├── backups/ | |
| 27 | - | ├── scan-spool/ | |
| 28 | - | └── git/ # GIT_REPOS_PATH (moved from /opt/git) | |
| 29 | - | ``` | |
| 30 | - | ||
| 31 | - | Systemd unit: | |
| 32 | - | ||
| 33 | - | ``` | |
| 34 | - | ExecStart=/opt/mnw/current/makenotwork | |
| 35 | - | WorkingDirectory=/opt/mnw/current | |
| 36 | - | EnvironmentFile=/etc/mnw/makenotwork.env | |
| 37 | - | ReadWritePaths=/var/lib/mnw /opt/mnw/yara-rules | |
| 38 | - | StateDirectory=mnw/scan-spool | |
| 39 | - | ``` | |
| 40 | - | ||
| 41 | - | ## Principle | |
| 42 | - | ||
| 43 | - | A sando "release" is the atomic versioned bundle in category #2 of the inventory below — code + version-coupled content. Anything that doesn't change in lockstep with the binary doesn't belong inside the release dir. Secrets and state live on paths sando never touches. This is what gives atomic rollback its actual value: one symlink swap moves every version-coupled thing together. | |
| 44 | - | ||
| 45 | - | ### The five categories (what prod actually serves) | |
| 46 | - | ||
| 47 | - | 1. **Versioned code** — `makenotwork`, `mnw-admin`. Tied to a git sha. | |
| 48 | - | 2. **Versioned content compiled-against-the-binary** — `static/` (cache-busted via compiled-in version), `error-pages/`, `docs/` (DocEngine content, askama-template-coupled). Skew between these and the binary is a bug class. | |
| 49 | - | 3. **Versioned content with its own update cadence** — `yara-rules/` (malware-scan rules, gated by SCAN_ENABLED). Updates independently of MNW releases. | |
| 50 | - | 4. **Secrets / config** — `.env`, 45 keys. Outlives every version; never in the repo. | |
| 51 | - | 5. **Runtime state** — `backups/`, `scan-spool/`, `git/`. Lives outside any version. | |
| 52 | - | ||
| 53 | - | Sando owns 1+2 as one atomic bundle. 3 stays operator-managed (revisit later if a real incident makes atomic yara-rules rollback necessary). 4 and 5 are not sando's concern. | |
| 54 | - | ||
| 55 | - | ## Session goal | |
| 56 | - | ||
| 57 | - | Sando produces a staged release dir with the full 1+2 bundle on the MM tier. No prod or testnot changes. Validate by `POST /rebuild` against MNW main and inspecting `/srv/sando/releases/<v>/`. | |
| 58 | - | ||
| 59 | - | ## A. Investigate first (~30 min) | |
| 60 | - | ||
| 61 | - | 1. `MNW/shared/docengine/` — find the build entry point. Is it a `cargo build -p docengine` producing a binary? A `build.rs` step in `server/`? A separate `docengine compile` command run over `site-docs/`? Determines whether sando's `build.rs` invokes cargo once or twice. If DocEngine compile is a cargo step, it falls out of the existing `cargo build --release`; if it's a separate binary that runs over `site-docs/`, sando needs to invoke it after the cargo build. | |
| 62 | - | 2. Confirm `server/static/` is real static (no preprocessing). Probably true — server/CONTRIBUTING.md notes "New JS goes in server/static/ as static files". | |
| 63 | - | 3. `mnw-admin` callers — grep `mnw-admin` across `server/` to confirm what invokes it. Per the user: build scripts + possibly the admin page. Those callers need to update to `/opt/mnw/current/mnw-admin` post-migration. Not a Session 1 change, just a note for Session 3. | |
| 64 | - | ||
| 65 | - | ## B. Sando code changes (`MNW/sando/daemon/src/`) | |
| 66 | - | ||
| 67 | - | ### `build.rs::build_and_run_mm` | |
| 68 | - | ||
| 69 | - | Currently stages `<release>/{makenotwork, mnw-admin, error-pages/}` via `deploy::deploy_local` (binaries) + a `cp -a` for error-pages. Extend to also stage: | |
| 70 | - | ||
| 71 | - | - `<release>/static/` ← `worktree/server/static/` (cp -a) | |
| 72 | - | - `<release>/docs/` ← `worktree/server/target/release/docs/` (assuming DocEngine outputs there) or `worktree/server/site-docs/` raw, depending on §A.1. | |
| 73 | - | ||
| 74 | - | Refactor: pull the per-asset `cp -a` calls into a single helper | |
| 75 | - | ||
| 76 | - | ```rust | |
| 77 | - | fn stage_dir(src: &Path, dst: &Path, required: bool) -> Result<()> { ... } | |
| 78 | - | ``` | |
| 79 | - | ||
| 80 | - | so the four sites (error-pages, static, docs, future) read uniformly. Missing-source policy: required=true errors; required=false logs warn and skips (so older shas without one of these don't break sando mid-bisect). | |
| 81 | - | ||
| 82 | - | ### `config.rs` | |
| 83 | - | ||
| 84 | - | Replace the `bin_names: Vec<String>` "primary binary" framing with two fields: | |
| 85 | - | ||
| 86 | - | ```rust | |
| 87 | - | pub bin_names: Vec<String>, // what cargo produces under target/release/ | |
| 88 | - | pub release_contents: Vec<ReleaseEntry>, // what gets staged from worktree → release dir | |
| 89 | - | ||
| 90 | - | pub struct ReleaseEntry { | |
| 91 | - | pub src: PathBuf, // relative to worktree root | |
| 92 | - | pub dst: PathBuf, // relative to release dir | |
| 93 | - | pub required: bool, | |
| 94 | - | } | |
| 95 | - | ``` | |
| 96 | - | ||
| 97 | - | Default value for MNW lives in `sando-daemon.toml`, not hard-coded in sando: | |
| 98 | - | ||
| 99 | - | ```toml | |
| 100 | - | release_contents = [ | |
| 101 | - | { src = "server/deploy/error-pages", dst = "error-pages", required = false }, | |
| 102 | - | { src = "server/static", dst = "static", required = true }, | |
| 103 | - | { src = "server/target/release/docs", dst = "docs", required = true }, | |
| 104 | - | ] | |
| 105 | - | ``` | |
| 106 | - | ||
| 107 | - | This pulls MNW-specific knowledge out of sando code into sando config — closer to what sando wants to be as a generic deploy controller. Tests can override with a fixture release_contents. | |
| 108 | - | ||
| 109 | - | ### `deploy.rs::deploy_node` | |
| 110 | - | ||
| 111 | - | Already rsyncs the whole staged dir, so no changes once `build.rs` stages more into it. Verify rsync flags include `-a` and `--delete` so removed assets don't accumulate across versions (e.g. a deleted static asset). | |
| 112 | - | ||
| 113 | - | ### `bootstrap-node.sh` | |
| 114 | - | ||
| 115 | - | Unit template now writes: | |
| 116 | - | ||
| 117 | - | ``` | |
| 118 | - | ExecStart=/opt/mnw/current/makenotwork | |
| 119 | - | WorkingDirectory=/opt/mnw/current | |
| 120 | - | EnvironmentFile=/etc/mnw/makenotwork.env | |
| 121 | - | ReadWritePaths=/var/lib/mnw /opt/mnw/yara-rules | |
| 122 | - | StateDirectory=mnw/scan-spool | |
| 123 | - | ``` | |
| 124 | - | ||
| 125 | - | Plus `install -d -o root -g $SERVICE_USER -m 0750 /etc/mnw /var/lib/mnw` upfront. Drop the `EnvironmentFile=-/opt/mnw/.env` default — env file lives at `/etc/mnw/makenotwork.env` now. | |
| 126 | - | ||
| 127 | - | ### Tier rename `mm` → `host` | |
| 128 | - | ||
| 129 | - | Folded in here while editing topology code. Schema migration in `sando/daemon/migrations/` renames the `tiers` row + any FK references in `tier_state`, `gate_runs`, `deploys`. Plus code sweep: `build.rs`, `routes.rs`, `sando.toml`, test fixtures. Add a defensive assert that loudly fails if `tier='mm'` is still queried anywhere — silent miss would be worse than a panic during the transition. | |
| 130 | - | ||
| 131 | - | ~30 lines of grep+replace + 1 sqlite migration. | |
| 132 | - | ||
| 133 | - | ## C. Test on MM | |
| 134 | - | ||
| 135 | - | 1. `cargo test --release --features fast-tests` in `sando/daemon/` — all existing tests pass + any new staging tests added (e.g. `stage_dir copies on success`, `stage_dir errors when required-missing`, `release_contents config parses`). | |
| 136 | - | 2. Build sandod, install, restart on fw13. | |
| 137 | - | 3. `POST /rebuild` against current MNW main. | |
| 138 | - | 4. Inspect `/srv/sando/releases/<v>/` — should contain `makenotwork`, `mnw-admin`, `error-pages/`, `static/`, `docs/`. Verify total size is sane (single-digit MB for binary, tens of MB for static+docs). | |
| 139 | - | 5. boot_smoke still passes (binary doesn't care what's in the dir alongside it). | |
| 140 | - | 6. cargo_test still green. | |
| 141 | - | ||
| 142 | - | ## D. Acceptance | |
| 143 | - | ||
| 144 | - | - A `/rebuild` against MNW main produces a staged release with all four asset categories (binaries, error-pages, static, docs). | |
| 145 | - | - Existing MM gates stay green. | |
| 146 | - | - `/srv/sando/releases/<v>/` is self-contained: deleting `/srv/sando/work/<sha>/` and re-running boot_smoke against the staged binary works. Sanity check that no path in the staged tree reaches back into the worktree. | |
| 147 | - | ||
| 148 | - | ## E. Out of scope for Session 1 (Session 2/3) | |
| 149 | - | ||
| 150 | - | - Any change to testnot or prod | |
| 151 | - | - Moving `/opt/git`, `/opt/makenotwork/.env`, state dirs | |
| 152 | - | - Authorizing sando's pubkey on prod | |
| 153 | - | - Renaming the systemd service path | |
| 154 | - | - yara-rules — stays operator-managed; out of sando entirely | |
| 155 | - | - Caddy config (operator-managed; uses bundle path indirectly via `localhost:3000` reverse_proxy + per-dir paths that point at `/opt/mnw/current/...` post-migration) | |
| 156 | - | ||
| 157 | - | ## F. Open questions to answer during Session 1 | |
| 158 | - | ||
| 159 | - | - **Total bundle size after staging.** If it's >50MB the rsync time per deploy gets noticeable. Worth measuring; not blocking. Sets expectations for Session 2/3. | |
| 160 | - | - **DocEngine build path.** Whether `cargo build --release` already produces `target/release/docs/` (if DocEngine is a build-script step) or needs an explicit `cargo run -p docengine -- build` (if it's a separate command). Determines whether `build.rs` in sando does one cargo invocation or two. | |
| 161 | - | - **DocEngine output format.** Is `target/release/docs/` a directory tree of HTML/assets ready to serve, or a single bundle file? Affects `stage_dir` semantics. | |
| 162 | - | ||
| 163 | - | ## Sessions 2 and 3 (out of scope but for context) | |
| 164 | - | ||
| 165 | - | **Session 2 — testnot migration (low-risk practice).** | |
| 166 | - | - `bootstrap-node.sh` on testnot with the new unit shape. | |
| 167 | - | - Write `/etc/mnw/makenotwork.env` on testnot from scratch — this is what was missing during the 2026-06-02 tier-A exercise attempt. | |
| 168 | - | - `POST /promote/a` from sando → boots green for the first time. | |
| 169 | - | - Exercise §6.5 step 3 tier-A flow we skipped. | |
| 170 | - | ||
| 171 | - | **Session 3 — prod migration (the careful one).** | |
| 172 | - | - Inventory + dry-run plan (most of inventory done 2026-06-02; one more pass for exact mv/install sequence). | |
| 173 | - | - Lock the deploy.sh path during the migration window. | |
| 174 | - | - Stop makenotwork.service. | |
| 175 | - | - `bootstrap-node.sh` on prod with `SANDO_PUBKEY=…`, creating `deploy` user + `/opt/mnw/` + new unit. | |
| 176 | - | - `mv /opt/makenotwork/.env /etc/mnw/makenotwork.env`. | |
| 177 | - | - `mv /opt/makenotwork/{backups,scan-spool}` → `/var/lib/mnw/` (or rebuild scan-spool — it's transient). | |
| 178 | - | - `mv /opt/git /var/lib/mnw/git` + update `GIT_REPOS_PATH` in `/etc/mnw/makenotwork.env`. | |
| 179 | - | - Audit all PATH-typed env keys against the new layout: `DOCS_PATH=/opt/mnw/current/docs`, `YARA_RULES_DIR=/opt/mnw/yara-rules`, `ASSUMPTIONS_PATH=/opt/mnw/current/docs/business/assumptions.toml`, etc. | |
| 180 | - | - Update build scripts on prod that invoke `mnw-admin` to use `/opt/mnw/current/mnw-admin`. | |
| 181 | - | - `POST /promote/b {"hotfix":true}` from sando → first sando deploy to prod. | |
| 182 | - | - Start makenotwork.service under the new layout. | |
| 183 | - | - Verify makenot.work end-to-end. | |
| 184 | - | - Soak for a week. | |
| 185 | - | - `rm -rf /opt/makenotwork/` after the soak; archive `deploy.sh` as break-glass-only. | |
| 186 | - | ||
| 187 | - | ## Key paths (for Claude orientation) | |
| 188 | - | ||
| 189 | - | - `MNW/sando/daemon/src/build.rs` — where staging happens (`build_and_run_mm`). | |
| 190 | - | - `MNW/sando/daemon/src/config.rs` — Config struct, `bin_names`. | |
| 191 | - | - `MNW/sando/daemon/src/deploy.rs` — `deploy_local`, `deploy_node`. | |
| 192 | - | - `MNW/sando/deploy/bootstrap-node.sh` — unit template. | |
| 193 | - | - `MNW/sando/sando.toml` — topology config (tier names live here). | |
| 194 | - | - `MNW/sando/daemon/sando-daemon.toml` — daemon config (release_contents will go here). | |
| 195 | - | - `MNW/sando/daemon/migrations/` — sqlite migrations for the mm→host rename. | |
| 196 | - | - `MNW/server/static/`, `MNW/server/site-docs/`, `MNW/server/deploy/error-pages/` — staging sources. | |
| 197 | - | - `MNW/shared/docengine/` — DocEngine crate (investigate in §A.1). | |
| 198 | - | - `launchplan_final.md` §6.5 — original tier-B decision context this redesign supersedes. | |
| 199 | - | - `MNW/sando/plans/config-artifacts.md` — earlier Phase 3 design doc on config vs binary artifacts; complementary background. | |
| 200 | - | ||
| 201 | - | ## G. Outcomes (2026-06-02) | |
| 202 | - | ||
| 203 | - | Session 1 landed in one focused push. All 10 tasks done, 44/44 sando-daemon tests green, pipeline went `host` green end-to-end against sha `f0970b8` (version 0.9.5) on fw13. | |
| 204 | - | ||
| 205 | - | ### What shipped (commit `f0970b8` on sando bare + mnw + srht remotes) | |
| 206 | - | ||
| 207 | - | - `release_contents: Vec<ReleaseEntry>` in `Config`, with `ReleaseEntry { src, dst, required }`. Sando code carries no MNW-specific knowledge; the MNW bundle shape lives in `sando-daemon.toml`. | |
| 208 | - | - `build.rs::build_and_run_host` (renamed from `_mm`) iterates `cfg.release_contents`, calling `stage_entry()` per row. `cp -a` semantics; supports the merge-into-existing-dir form so multiple entries can target the same `dst` (used for `docs/` from 3 worktree sources). | |
| 209 | - | - `deploy.rs` rsync gained `--delete` (no stale assets across versions) and swapped `--chmod=F0755` for `--chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r,F+X` (binaries 0755, data files 0644). | |
| 210 | - | - `bootstrap-node.sh` writes FHS-style unit: `EnvironmentFile=/etc/mnw/makenotwork.env`, `ReadWritePaths=/var/lib/mnw`, `WorkingDirectory=<release>/current`. Pre-creates `/etc/mnw` (root:service 0750) + `/var/lib/mnw` (service:service 0750). | |
| 211 | - | - Migration `002_rename_mm_to_host.sql` — `PRAGMA defer_foreign_keys = ON` + 5 UPDATEs (tiers, nodes, deploys, gate_runs, tier_state). Preserved all existing state on fw13 (host current=0.9.5 + a current=0.8.12 carried through). | |
| 212 | - | - Post-receive hook now lives in repo at `sando/deploy/post-receive` and sources `/etc/sando/sando.env` — `SANDO_DAEMON` resolves to the tailnet listener instead of the 127.0.0.1 default. `bootstrap-sandod-host.sh` installs it. | |
| 213 | - | ||
| 214 | - | ### Open-question answers from §F | |
| 215 | - | ||
| 216 | - | - **Bundle size:** 154 MB total. 133 MB makenotwork + 25 MB mnw-admin + <1 MB of error-pages/static/docs combined. Non-binary content is rounding error against the binaries; rsync over the LAN+tailnet to testnot/prod will be dominated by binary delta (rsync's algorithm helps here — minor code changes ship as small deltas, not 158 MB). | |
| 217 | - | - **DocEngine build path:** It's a library crate (`MNW/shared/docengine/`), not a binary. No separate build step. Content stays raw markdown at `server/site-docs/` + `server/docs/business/assumptions.toml`. Three `release_contents` rows merge them into one `docs/` dir. | |
| 218 | - | - **DocEngine output format:** N/A — content never compiles. Raw `.md` files; the running server reads them at request time via the env-configured paths. | |
| 219 | - | ||
| 220 | - | ### Surprises / unplanned discoveries | |
| 221 | - | ||
| 222 | - | - **deploy.sh has a CSS minification step** (`npx clean-css-cli`) before rsync. Sando does not. Effect: bundle ships unminified CSS (~3x larger on the wire than deploy.sh-shipped CSS). `server/build.rs` hashes the *unminified* `style.css` for the cache-bust `?v=...`, so correctness is preserved — purely a size issue. Future fix: either eat the size cost (gzip handles most of it), move minification into `server/build.rs`, or add a build-step gate to sando. **Not addressed in Session 1.** | |
| 223 | - | - **mnw-admin invocation surface is bigger than expected.** Live call sites on prod: (1) sudoers `/etc/sudoers.d/*` entry `makenotwork ALL=(git) NOPASSWD: /opt/makenotwork/mnw-admin rebuild-keys` — needs path update in Session 3; (2) `command=` prefixes in `/home/git/.ssh/authorized_keys` that `mnw-admin rebuild-keys` itself generates — auto-update on the first post-migration rebuild-keys run. Session 3 sequence: edit the sudoers file first, then run `mnw-admin rebuild-keys` once. | |
| 224 | - | - **The defensive assert-on-stray-`"mm"`-lookup proposed in the plan was skipped.** Tests catch it: any unrenamed site fails when the DB no longer has a row matching it. After the rename + sync.rs test run + the production restart on fw13, no "mm" lookups remained. | |
| 225 | - | ||
| 226 | - | ### Carry-over for Session 2 | |
| 227 | - | ||
| 228 | - | The Session 2 starting point shifted slightly because we did Session 1's prep + ran it in one push. State to assume when Session 2 begins: | |
| 229 | - | ||
| 230 | - | - `f0970b8` is the active sha on sando's bare repo and is current on tier host. Tier a is on the stale `0.8.12` from pre-Session-1. | |
| 231 | - | - testnot still has the unit shape from the *pre-Session-1* `bootstrap-node.sh`. It is in a crashloop (MissingDatabaseUrl, no env file). Session 2 reprovisioning will replace its systemd unit with the new FHS shape AND populate `/etc/mnw/makenotwork.env` from scratch. | |
| 232 | - | - `bootstrap-sandod-host.sh` on fw13 is the new version; re-running it is idempotent. | |
| 233 | - | - Sando's pubkey on testnot under the `deploy` user: confirmed working earlier (`sudo -u sando ssh deploy@testnot` returned). No re-auth needed. | |
| 234 | - | - The bundle has not yet been deployed remotely. Tier a's `0.8.12` deploy predates `release_contents`; the testnot release dir contains only binaries. First Session 2 promotion to tier a will be the first remote deploy of the full bundle. | |
| 235 | - | ||
| 236 | - | ### Things Session 2 should re-check before promoting | |
| 237 | - | ||
| 238 | - | - Verify `/etc/mnw/` and `/var/lib/mnw/` get created with the right ownership when `bootstrap-node.sh` runs on testnot (the new code path; never tested on a real box). | |
| 239 | - | - Decide what testnot's env file looks like — full prod-clone (with real Stripe test keys) or minimal (just what `Config::from_env` requires to boot). Minimal is faster and validates the deploy path; prod-clone exercises more code paths. | |
| 240 | - | - Confirm sando's pubkey is in `/home/deploy/.ssh/authorized_keys` on testnot, not just routable via Tailscale SSH. (Tailscale SSH ≠ pubkey auth; sando uses pubkey-only via OpenSSH.) | |
| 241 | - |
| @@ -1,163 +0,0 @@ | |||
| 1 | - | # Session 3 — first sando-driven prod deploy | |
| 2 | - | ||
| 3 | - | Captured 2026-06-03 after the cutover. Resolves §6.5 step 8 of `launchplan_final.md`: first full sando deploy to Hetzner prod, replacing `deploy.sh` as the live deploy path. | |
| 4 | - | ||
| 5 | - | Status: **complete 2026-06-03.** Prod runs `makenotwork` 0.9.5 (sha `f0970b8`) from `/opt/mnw/current/`, deployed via `POST /promote/b {"hotfix":true}` from sandod on fw13. Outage window 3m25s (02:50:33 → 02:53:58 UTC). All features green. See §F for outcomes and §G for the four hardcoded paths that block the eventual `rm -rf /opt/makenotwork/`. | |
| 6 | - | ||
| 7 | - | ## Background — Session 1 set the layout, Session 2 proved it on testnot, Session 3 cut prod over | |
| 8 | - | ||
| 9 | - | Session 1 redesigned the on-disk layout (`/opt/mnw/releases/<v>/` + `current` symlink; `/etc/mnw/makenotwork.env`; `/var/lib/mnw/` for state) and shipped the sando-side code that produces the full versioned bundle (binaries + static + docs + error-pages + assumptions). Session 2 reprovisioned testnot under that layout; the first remote deploy of the full bundle landed cleanly after three small gotchas (sqlx URL form, pg_ident map, `ASSUMPTIONS_PATH` mismatch — all logged in `launchplan_final.md` §6.9). | |
| 10 | - | ||
| 11 | - | Session 3 is the real-stakes one: prod was on 0.9.1 via `deploy.sh`, `/opt/makenotwork/` had eight months of accreted state (885M of backups, .env, yara-rules, ssh dir, rustdoc, sudoers entries, cron jobs, Caddyfile references). The Session 1 plan enumerated some of the move sequence but understated the surface area; the actual cutover surfaced several things worth documenting so the next major reprovision (or a disaster-recovery rebuild) doesn't re-discover them. | |
| 12 | - | ||
| 13 | - | ## A. Inventory taken before any prod write | |
| 14 | - | ||
| 15 | - | `/opt/makenotwork/` contents (`makenotwork:makenotwork` unless noted): | |
| 16 | - | ||
| 17 | - | - `makenotwork`, `mnw-admin` — 0.9.1 binaries (`root:root`) | |
| 18 | - | - `.env` (110 lines), 5× `.env.bak.*` files (`root:root`) | |
| 19 | - | - `docs/`, `static/`, `error-pages/` — content (will be replaced by release bundle) | |
| 20 | - | - `backups/` — 885M | |
| 21 | - | - `yara-rules/` — 8.5M compiled, `root:root` | |
| 22 | - | - `yara-rules-src/` — upstream YARA sources (compiled to `yara-rules/`), `root:root` | |
| 23 | - | - `rustdoc/` — generated docs, `501:staff` (uploaded from Mac via `deploy.sh`) | |
| 24 | - | - `ssh/` — `known_hosts` for build runner, `root:root` | |
| 25 | - | - `backup-db.sh` — cron'd daily at 03:00 UTC from `makenotwork`'s crontab | |
| 26 | - | - `deploy/` — `deploy.sh` staging area, `root:root` | |
| 27 | - | ||
| 28 | - | Other prod state in play: | |
| 29 | - | ||
| 30 | - | - `/opt/git/` — 99M, `git:git`. Both git user's home (`/etc/passwd` says `git:x:995:986::/opt/git:/bin/sh`) *and* the GIT_REPOS_PATH target. Conflating these turns out to matter (§F). | |
| 31 | - | - `/etc/caddy/Caddyfile` — three `root * /opt/makenotwork/error-pages` lines. | |
| 32 | - | - `/etc/sudoers.d/mnw-git-ssh` — `makenotwork ALL=(git) NOPASSWD: /opt/makenotwork/mnw-admin rebuild-keys`. | |
| 33 | - | - `/etc/sudoers.d/mnw-cli-git` — `mnw-cli ALL=(git) NOPASSWD: /usr/bin/git-*, /usr/bin/tee, /usr/bin/chmod`. No /opt path references; left alone. | |
| 34 | - | - `makenotwork` user crontab: `0 3 * * * /opt/makenotwork/backup-db.sh >> /opt/makenotwork/backups/backup.log 2>&1`. | |
| 35 | - | - Root crontab: `0 3 * * * /opt/backups/pg_backup.sh >> /var/log/pg_backup.log 2>&1` — unrelated, left alone. | |
| 36 | - | ||
| 37 | - | ## B. Pre-flight (no prod impact) | |
| 38 | - | ||
| 39 | - | 1. **`sando.toml` tier B fixed.** Was `deploy@prod-1.makenot.work` (NXDOMAIN, no port). Now `makenotwork@alpha-west-1` with port handling via `~sando/.ssh/config` Host block. Chose to keep service user as `makenotwork` rather than introduce a `deploy` user — avoids chowning 885M of backups and redoing pg peer auth that's been stable for months. The same reasoning applies to a hypothetical tier C: keep the existing user, don't introduce a new one for cosmetic uniformity with testnot. | |
| 40 | - | 2. **Sando pubkey installed** in `/home/makenotwork/.ssh/authorized_keys` (mode 0600, owned makenotwork). | |
| 41 | - | 3. **`chsh -s /bin/bash makenotwork`** — was `/usr/sbin/nologin`. SSH was rejecting connections, not key auth failing. Worth detecting/fixing in `bootstrap-node.sh` for future provisions where someone has hardened the runtime user. | |
| 42 | - | 4. **`/srv/sando/.ssh/config`** Host block for port 2200; `known_hosts` seeded via `ssh-keyscan -p 2200`. | |
| 43 | - | 5. **Dry-run rsync** from sando → prod's `/opt/mnw/releases/_probe/` succeeded (after `bootstrap-node.sh` created `/opt/mnw/`). | |
| 44 | - | ||
| 45 | - | ## C. Cutover sequence (3m25s outage) | |
| 46 | - | ||
| 47 | - | In order, with the exact reason each step exists: | |
| 48 | - | ||
| 49 | - | 1. **`systemctl stop makenotwork`** — 02:50:33 UTC. Outage window starts. | |
| 50 | - | 2. **Backups taken**: `/etc/systemd/system/makenotwork.service → /root/makenotwork.service.bak-pre-cutover`; `/opt/makenotwork/.env → /root/dotenv.bak-pre-cutover`; Caddyfile, sudoers, crontab also backed up to `/root/*.bak-pre-cutover`. Rollback path for any step failing before service restart. | |
| 51 | - | 3. **`bootstrap-node.sh`** with `SERVICE_USER=makenotwork SANDO_PUBKEY=… INSTALL_POSTGRES=0 INSTALL_CADDY=0 INSTALL_TAILSCALE=0 ENABLE_FIREWALL=0` — postgres/caddy/tailscale/UFW already configured on prod, don't touch. Created `/opt/mnw/`, `/etc/mnw/`, `/var/lib/mnw/`, the new systemd unit, the unused `deploy` user (harmless), the sudoers entry for `deploy`. The new unit references `EnvironmentFile=/etc/mnw/makenotwork.env` and `ReadWritePaths=/var/lib/mnw`, with `RestartPreventExitStatus=2` (MNW server convention: exit 2 = migration failure, don't crashloop). | |
| 52 | - | 4. **`cp /opt/makenotwork/.env /etc/mnw/makenotwork.env`** (copy, not move — original stays for one-week rollback). `chmod 0640 root:makenotwork`. Then `sed` rewrites of `DOCS_PATH`, `ASSUMPTIONS_PATH`, `YARA_RULES_DIR`, `GIT_REPOS_PATH` for the new layout. `HOST`, `PORT`, `DATABASE_URL`, `HOST_URL` unchanged. | |
| 53 | - | 5. **`ln -s /opt/makenotwork/yara-rules /opt/mnw/yara-rules`** — yara-rules is operator-managed (independent update cadence), not in the release bundle (Session 1 layout principle: category #3). The symlink lets the new env's `YARA_RULES_DIR=/opt/mnw/yara-rules` continue to resolve. When `/opt/makenotwork/` is eventually removed, the rules dir moves to a permanent path (probably `/var/lib/mnw/yara-rules` or `/etc/mnw/yara-rules`) and the symlink retargets. | |
| 54 | - | 6. **`rsync -aHX /opt/git/ /var/lib/mnw/git/`** — preserves `git:git` ownership and the directory hardlinks. `chmod 0755 /var/lib/mnw` so the git user can traverse (default was 0750 makenotwork:makenotwork, which blocked git's git-receive-pack from reaching the repos). | |
| 55 | - | 7. **Caddyfile rewrite**: `sed -i 's|/opt/makenotwork/error-pages|/opt/mnw/current/error-pages|g'`. `caddy validate` before reload; `systemctl reload caddy`. | |
| 56 | - | 8. **Sudoers rewrite**: same sed pattern on `/etc/sudoers.d/mnw-git-ssh`; `visudo -c -f` to validate. | |
| 57 | - | 9. **`systemctl daemon-reload`** to pick up the new unit. | |
| 58 | - | 10. **`systemctl restart sandod`** on fw13 — sandod caches `sando.toml` at startup; the new tier B target wouldn't have taken effect without this. **First `POST /promote/b` failed with NXDOMAIN against the stale `prod-1.makenot.work` because sandod hadn't been restarted yet.** Fixed by restarting sandod and re-promoting. | |
| 59 | - | 11. **`POST /promote/b {"hotfix":true}`** — `hotfix: true` bypasses the 48h burn-in on tier A (which had just promoted to 0.9.5 ~15 min prior; burn-in not yet elapsed). Sando rsync'd the 161MB bundle to `/opt/mnw/releases/0.9.5/`, swapped the `current` symlink, called `systemctl reload-or-restart makenotwork.service`. | |
| 60 | - | 12. **Service up 02:53:55 UTC.** Outage window ends 02:53:58 once health serves 200. 733 YARA rules compiled, all integrations (S3, Stripe, MT, WAM, git, scanner, custom domain cache) live. | |
| 61 | - | 13. **External smoke checks**: `/`, `/login`, `/pricing`, `/docs`, `/docs/economics`, `/docs/roadmap`, `/docs/tiers` — all 200. | |
| 62 | - | 14. **`rebuild-keys` to regenerate `/opt/git/.ssh/authorized_keys`** — `dotenvy` doesn't auto-load when running mnw-admin standalone (it loads from `/opt/makenotwork/.env`, mode 0600 `makenotwork:makenotwork`, unreadable by git). Worked around by sourcing the env in root then `sudo -u git -E`. **Regenerated keys still contain `command="/opt/makenotwork/mnw-admin git-auth ..."`** — see §G. | |
| 63 | - | 15. **Git push test** — `git ls-remote git@ssh.makenot.work:max/meta.git` returns refs cleanly. Cutover verified end-to-end. | |
| 64 | - | ||
| 65 | - | ## D. What stayed in place (intentional) | |
| 66 | - | ||
| 67 | - | - `/opt/makenotwork/` — full contents, untouched. Soak rollback path: stop new unit, swap systemd unit back, start old binary. Plan: `rm -rf` after a week, post-0.9.6 deploy (see §G). | |
| 68 | - | - `/opt/git/` — untouched. Git user's `/etc/passwd` home; mnw-admin's regenerated `authorized_keys` writes to `/opt/git/.ssh/authorized_keys` (not `/home/git/`, despite earlier confusion). The rsync to `/var/lib/mnw/git/` populated the new GIT_REPOS_PATH; the server reads from there, but git push lands in `/opt/git/` because that's git user's home. Both paths now hold the repo bytes; that's wasteful but harmless during the soak. | |
| 69 | - | - `/opt/makenotwork/backups/` — 885M of pg dumps. Script and cron still write there. Sando's backup-fetch on fw13 still pulls from there (configured pre-cutover). Migration to `/var/lib/mnw/backups/` is its own follow-up (touches script, crontab, fw13 sando config). | |
| 70 | - | - `yara-rules-src/`, `rustdoc/`, `ssh/`, `.env.bak.*` — not in any env var or systemd path. Confirmed by grepping the running 0.9.5 binary's path references. Will be swept in the post-soak cleanup. | |
| 71 | - | ||
| 72 | - | ## E. What broke and how it was caught | |
| 73 | - | ||
| 74 | - | Three small things, all caught by smoke checks: | |
| 75 | - | ||
| 76 | - | 1. **`sandod` cached `sando.toml`.** First promote attempt returned `creating remote release dir` (an in-flight progress string that became the error message). `journalctl -u sandod` showed it was still resolving `prod-1.makenot.work`. `scp sando.toml fw13:/tmp/`, `sudo cp /tmp/sando.toml /etc/sando/sando.toml`, `sudo systemctl restart sandod`, re-promote. Worth documenting that `sandod` does not watch the file; alternative is to add an inotify or SIGHUP handler. | |
| 77 | - | 2. **First doc smoke checks were wrong URLs.** `/about/economics`, `/docs/about/economics` returned 404; panicked briefly that the cutover broke doc routing. False alarm: the route is `/docs/{slug}` where slug is the filename stem (e.g., `/docs/economics`). Verified with `grep doc_page MNW/server/src/` after the panic. **Worth fixing in any future smoke script** — use the real URL scheme, not guessed-from-filesystem paths. | |
| 78 | - | 3. **`mnw-admin rebuild-keys` needed env loading from root context.** `sudo -u git /opt/mnw/current/mnw-admin rebuild-keys` fails with `DATABASE_URL must be set: NotPresent` because the binary's `dotenvy::from_path("/opt/makenotwork/.env")` runs as git, which can't read `.env` (mode 0600 makenotwork). Workaround: `set -a; source /etc/mnw/makenotwork.env; set +a; sudo -u git -E /opt/mnw/current/mnw-admin rebuild-keys`. Cleanest long-term fix is in §G. | |
| 79 | - | ||
| 80 | - | ## F. Outcomes (verified) | |
| 81 | - | ||
| 82 | - | **Sando state after cutover:** | |
| 83 | - | ||
| 84 | - | ``` | |
| 85 | - | host cur=0.9.5 prev=0.9.5 burn_in_started=2026-06-03T02:23:28Z | |
| 86 | - | a cur=0.9.5 prev=0.8.12 burn_in_started=2026-06-03T02:38:57Z | |
| 87 | - | b cur=0.9.5 prev=None burn_in_started=2026-06-03T02:53:56Z | |
| 88 | - | c not provisioned | |
| 89 | - | ``` | |
| 90 | - | ||
| 91 | - | **Prod externally:** | |
| 92 | - | - `https://makenot.work/api/health` → `{"status":"operational","version":"0.9.5","checks":{"database":true}}`. | |
| 93 | - | - `/`, `/login`, `/pricing`, `/docs`, `/docs/economics`, `/docs/roadmap`, `/docs/tiers` → 200. | |
| 94 | - | - Git: `git ls-remote git@ssh.makenot.work:max/meta.git` → returns refs. | |
| 95 | - | ||
| 96 | - | **Prod internally:** | |
| 97 | - | - `systemctl status makenotwork` → active, PID 3123111, listening 0.0.0.0:3000. | |
| 98 | - | - 733 YARA rules compiled from `/opt/mnw/yara-rules` (symlink). | |
| 99 | - | - All integrations enabled per startup log: `s3=true, synckit_s3=false, stripe=true, scanner=true, mt=true, wam=true, git=true`. | |
| 100 | - | ||
| 101 | - | **deploy.sh path retained.** Not retired; remains as break-glass per `feedback_prefer_sando_over_deploy_sh` (sando is preferred *default*; deploy.sh stays runnable for outages where sando host is down). | |
| 102 | - | ||
| 103 | - | ## G. Open follow-ups | |
| 104 | - | ||
| 105 | - | ### G.1 The hardcoded `/opt/makenotwork/` paths (blocks the cleanup milestone) | |
| 106 | - | ||
| 107 | - | Session 1 outcomes claimed "`command=` prefixes auto-update on the first post-migration `rebuild-keys` run." That's wrong — confirmed during step 14. The path is a `const` in the binary, not pulled from env. Four sites need lifting before `/opt/makenotwork/` can be removed: | |
| 108 | - | ||
| 109 | - | | File | Line | Current value | Target | | |
| 110 | - | |---|---|---|---| | |
| 111 | - | | `server/src/git_ssh.rs` | 15 | `const MNW_ADMIN_PATH: &str = "/opt/makenotwork/mnw-admin"` | `/opt/mnw/current/mnw-admin` | | |
| 112 | - | | `server/src/bin/mnw-admin.rs` | 122 | `dotenvy::from_path("/opt/makenotwork/.env")` | `/etc/mnw/makenotwork.env` | | |
| 113 | - | | `server/src/build_runner.rs` | 467 | `const BUILD_SSH_KNOWN_HOSTS: &str = "/opt/makenotwork/ssh/known_hosts"` | `/etc/mnw/known_hosts` (or delete if dead — verify usage first) | | |
| 114 | - | | `server/src/routes/api/ssh_keys.rs` | 165 | `args(["-u", "git", "/opt/makenotwork/mnw-admin", "rebuild-keys"])` | `/opt/mnw/current/mnw-admin` | | |
| 115 | - | ||
| 116 | - | Ship as 0.9.6. Cleanup sequence after: deploy 0.9.6 via sando → `rebuild-keys` once (regenerates `authorized_keys` with new path in command=) → soak one week → `rm -rf /opt/makenotwork/`. | |
| 117 | - | ||
| 118 | - | ### G.2 The backups dir migration | |
| 119 | - | ||
| 120 | - | Independent of G.1. Touches: | |
| 121 | - | - `server/deploy/backup-db.sh` — hardcoded `BACKUP_DIR="/opt/makenotwork/backups"` near top. | |
| 122 | - | - `makenotwork` user crontab on prod. | |
| 123 | - | - Sando's `backup.source` URL on fw13 (currently pulls from `/opt/makenotwork/backups/latest.sql.gz` via rrsync). | |
| 124 | - | ||
| 125 | - | Easiest order: copy the existing 885M dir to `/var/lib/mnw/backups/`, edit script + crontab + sando config in one window, retire `/opt/makenotwork/backups/` after one successful daily backup lands in the new location and sando confirms it pulled cleanly. | |
| 126 | - | ||
| 127 | - | ### G.3 The `/opt/git` vs `/var/lib/mnw/git` duality | |
| 128 | - | ||
| 129 | - | Both directories currently hold the same repos. Git pushes land in `/opt/git/` (git user's home from `/etc/passwd`). Server reads from `/var/lib/mnw/git/` (GIT_REPOS_PATH). They drift the moment someone pushes. | |
| 130 | - | ||
| 131 | - | Two ways out: | |
| 132 | - | - (a) `usermod -d /var/lib/mnw/git git` to make git's home match GIT_REPOS_PATH. Single source of truth. Risk: any cron / script that reads git's home (none I found, but worth grepping) breaks. | |
| 133 | - | - (b) Revert GIT_REPOS_PATH to `/opt/git/`. Avoids the move but locks the path forever and reverts a piece of Session 1's FHS migration. | |
| 134 | - | ||
| 135 | - | (a) is the right answer. Do it during the post-0.9.6 soak window. | |
| 136 | - | ||
| 137 | - | ### G.4 `bootstrap-node.sh` polish | |
| 138 | - | ||
| 139 | - | From this cutover and Session 2: | |
| 140 | - | ||
| 141 | - | - **Detect `nologin` shell** on `SERVICE_USER` and refuse with a clear error (or auto-`chsh`). Costs ~1 min of cutover time if you don't know to check. | |
| 142 | - | - **Sibling `bootstrap-node-postgres.sh`** for the common pg_ident map case (when SERVICE_USER ≠ pg role name). Or document the manual steps in the script's "next steps" output. | |
| 143 | - | - **README-postgres.md note** on the sqlx URL form: `postgres:///db?host=/var/run/postgresql&user=name`, not `postgres://user@/db?host=...`. | |
| 144 | - | ||
| 145 | - | ### G.5 `ASSUMPTIONS_PATH` mismatch | |
| 146 | - | ||
| 147 | - | `sando-daemon.toml` puts the file at `<release>/docs/assumptions.toml`; prod's pre-existing env expected `<release>/docs/business/assumptions.toml` (matching the source layout `server/docs/business/assumptions.toml`). Worked around with an env edit during cutover but both prod and testnot now have non-canonical `ASSUMPTIONS_PATH=/opt/mnw/current/docs/assumptions.toml`. Fix: change `release_contents[3].dst` in `sando-daemon.toml` to `docs/business/assumptions.toml` and revert the env path on both nodes. Small, do it during the 0.9.6 sprint. | |
| 148 | - | ||
| 149 | - | ## H. Key paths (for orientation) | |
| 150 | - | ||
| 151 | - | - `MNW/sando/sando.toml` — tier B definition (`makenotwork@alpha-west-1`). | |
| 152 | - | - `MNW/sando/deploy/bootstrap-node.sh` — node-bootstrap; ran on prod with `SERVICE_USER=makenotwork`. | |
| 153 | - | - `MNW/sando/daemon/sando-daemon.toml` — release_contents (note §G.5 ASSUMPTIONS_PATH mismatch). | |
| 154 | - | - `MNW/server/src/{git_ssh.rs, build_runner.rs, bin/mnw-admin.rs, routes/api/ssh_keys.rs}` — the four hardcoded path sites. | |
| 155 | - | - `MNW/server/deploy/backup-db.sh` — hardcoded backup dir. | |
| 156 | - | - `/etc/systemd/system/makenotwork.service` (prod) — new FHS unit. | |
| 157 | - | - `/etc/mnw/makenotwork.env` (prod) — new env file location. | |
| 158 | - | - `/etc/sudoers.d/mnw-git-ssh` (prod) — updated to `/opt/mnw/current/mnw-admin`. | |
| 159 | - | - `/etc/caddy/Caddyfile` (prod) — three error-pages refs updated. | |
| 160 | - | - `/opt/makenotwork/` (prod) — full pre-cutover state, kept for soak rollback. | |
| 161 | - | - `launchplan_final.md` §6.5 step 8 — original plan this session closes. | |
| 162 | - | - `launchplan_final.md` §6.9 — Session 2/3 gotchas summary. | |
| 163 | - | - `launchplan_final.md` §7 — 0.9.6 path-decoupling spec. |