| 1 |
# CI/CD Pipeline |
| 2 |
|
| 3 |
## Overview |
| 4 |
|
| 5 |
Push-triggered CI on astra, WAM ticket notifications, staged promotion to Hetzner production. Monitored via a ratatui TUI on the tailnet. |
| 6 |
|
| 7 |
## Architecture |
| 8 |
|
| 9 |
``` |
| 10 |
MacBook (dev) |
| 11 |
| |
| 12 |
| git push mnw main |
| 13 |
v |
| 14 |
Hetzner (production, x86_64) |
| 15 |
| |
| 16 |
| post-receive hook: ssh astra |
| 17 |
v |
| 18 |
Astra (CI/build, aarch64 Linux) |
| 19 |
| |
| 20 |
| 1. cargo check |
| 21 |
| 2. cargo test --lib |
| 22 |
| 3. cargo test --test integration |
| 23 |
| 4. cargo clippy |
| 24 |
| 5. cargo audit |
| 25 |
| |
| 26 |
| on success: |
| 27 |
| - WAM ticket (pass, test count, duration) |
| 28 |
| - cross-compile x86_64 binary (cargo-zigbuild) |
| 29 |
| - scp binary to Hetzner staging dir |
| 30 |
| |
| 31 |
| on failure: |
| 32 |
| - WAM ticket (fail, which step, error) |
| 33 |
| - stop (do not build or promote) |
| 34 |
| |
| 35 |
v |
| 36 |
Hetzner staging (/opt/makenotwork-staging/) |
| 37 |
| |
| 38 |
| PoM health checks staging instance on port 3001 |
| 39 |
| |
| 40 |
| manual or TUI-driven promote |
| 41 |
v |
| 42 |
Hetzner production (/opt/makenotwork/) |
| 43 |
``` |
| 44 |
|
| 45 |
## Machines |
| 46 |
|
| 47 |
|
| 48 |
|
| 49 |
| MacBook | Development | aarch64 macOS | local | |
| 50 |
| Astra | CI runner, native aarch64 + cross-compile x86_64 | aarch64 Linux | 100.106.221.39 | |
| 51 |
| Hetzner | Production server, staging host | x86_64 Linux | 100.120.174.96 | |
| 52 |
|
| 53 |
Astra is aarch64 Linux. It runs tests natively and cross-compiles for x86_64 (Hetzner) via cargo-zigbuild. |
| 54 |
|
| 55 |
--- |
| 56 |
|
| 57 |
## Step 1: Push-triggered CI -- IMPLEMENTED (pending astra SSH) |
| 58 |
|
| 59 |
**Scripts:** |
| 60 |
- `deploy/post-receive-hook.sh` -- git hook for Hetzner (triggers CI on main push, OTA on tag push) |
| 61 |
- `deploy/ci-on-push.sh` -- wrapper on astra (pulls, runs CI, reports to WAM) |
| 62 |
- `deploy/setup-ci.sh` -- one-time provisioning (SSH key, hook install) |
| 63 |
|
| 64 |
**Trigger:** Git post-receive hook on Hetzner SSHes to astra on `refs/heads/main` push. |
| 65 |
|
| 66 |
**What runs:** `/home/max/staging/ci-on-push.sh` calls existing `run-ci.sh`: |
| 67 |
- cargo check |
| 68 |
- cargo test --lib (unit tests) |
| 69 |
- cargo test --test integration (8 threads, isolated per-test databases) |
| 70 |
- cargo clippy |
| 71 |
- cargo audit |
| 72 |
|
| 73 |
**Notification:** WAM ticket created with pass/fail, step counts, duration, last 30 lines of output. |
| 74 |
|
| 75 |
**Setup required:** |
| 76 |
1. Generate SSH key for git user on Hetzner (`deploy/setup-ci.sh` handles this) |
| 77 |
2. Install public key on astra `~/.ssh/authorized_keys` |
| 78 |
3. Copy `ci-on-push.sh` to astra `/home/max/staging/` |
| 79 |
4. Install hook on Hetzner (setup script handles this) |
| 80 |
5. Astra needs: `WAM_URL=http://100.120.174.96:7890` in environment |
| 81 |
|
| 82 |
--- |
| 83 |
|
| 84 |
## Step 2: Auto-build on CI success |
| 85 |
|
| 86 |
On CI pass, `ci-on-push.sh` continues: |
| 87 |
1. Cross-compile x86_64 release binary on astra (`cargo zigbuild --release --target x86_64-unknown-linux-gnu`) |
| 88 |
2. scp binary to Hetzner `/opt/makenotwork-staging/makenotwork` |
| 89 |
3. WAM ticket: "build ready for staging" |
| 90 |
|
| 91 |
Astra prerequisites: |
| 92 |
- `rustup target add x86_64-unknown-linux-gnu` |
| 93 |
- `cargo install cargo-zigbuild` (or system zig + zigbuild) |
| 94 |
|
| 95 |
--- |
| 96 |
|
| 97 |
## Step 3: Staging environment |
| 98 |
|
| 99 |
Hetzner runs a second instance for validation before production: |
| 100 |
- Binary at `/opt/makenotwork-staging/makenotwork` |
| 101 |
- Systemd unit: `makenotwork-staging.service` (port 3001) |
| 102 |
- Database: `makenotwork_staging` (separate from production) |
| 103 |
- Env: copy of production `.env` with different port and DB URL |
| 104 |
- PoM health-checks staging at `http://100.120.174.96:3001/api/health` |
| 105 |
- WAM ticket on staging health pass: "staging healthy, ready to promote" |
| 106 |
|
| 107 |
--- |
| 108 |
|
| 109 |
## Step 4: Promote to production |
| 110 |
|
| 111 |
`deploy/promote.sh`: |
| 112 |
1. Send 30s restart warning via internal API |
| 113 |
2. Stop production service |
| 114 |
3. Copy staging binary to `/opt/makenotwork/makenotwork` |
| 115 |
4. Start production service |
| 116 |
5. PoM verifies health within 30s |
| 117 |
6. If unhealthy: rollback (restore previous binary from `/opt/makenotwork/makenotwork.prev`, restart) |
| 118 |
7. WAM ticket: promoted or rolled back |
| 119 |
|
| 120 |
--- |
| 121 |
|
| 122 |
## Step 5: CI/CD Monitor TUI |
| 123 |
|
| 124 |
A ratatui-based TUI (part of WAM or standalone) for real-time pipeline visibility across the tailnet. Connects to WAM's HTTP API to display CI/CD state. |
| 125 |
|
| 126 |
### Views |
| 127 |
|
| 128 |
**Pipeline dashboard (default):** |
| 129 |
``` |
| 130 |
CI/CD v0.4.0 -> v0.4.1 |
| 131 |
--------------------------------------------------------------- |
| 132 |
[v] check [v] test-lib [v] test-int [v] clippy [v] audit |
| 133 |
[v] build [v] staging [ ] promote |
| 134 |
--------------------------------------------------------------- |
| 135 |
Last CI: 2m ago (42s, 715 tests passed) |
| 136 |
Staging: healthy (3m uptime) |
| 137 |
Prod: v0.4.0 (2h uptime) |
| 138 |
``` |
| 139 |
|
| 140 |
**Recent runs:** |
| 141 |
- List of recent CI runs with pass/fail, duration, commit hash |
| 142 |
- Filter by branch, status |
| 143 |
- Enter to view full log |
| 144 |
|
| 145 |
**Actions (keyboard-driven):** |
| 146 |
- `p` -- promote staging to production (with confirmation) |
| 147 |
- `r` -- re-run CI manually (SSH to astra) |
| 148 |
- `s` -- view staging health details |
| 149 |
- `l` -- view full CI log |
| 150 |
- `d` -- deploy current staging binary (skip CI) |
| 151 |
|
| 152 |
**Data sources:** |
| 153 |
- WAM tickets (source: `ci`) for CI run results |
| 154 |
- WAM tickets (source: `build-failed`, `health-status-change`) for problems |
| 155 |
- PoM API for staging + production health status |
| 156 |
- SSH to astra for live CI log streaming (if a run is in progress) |
| 157 |
|
| 158 |
### Implementation |
| 159 |
|
| 160 |
Could be: |
| 161 |
1. A new `wam pipeline` subcommand (simplest, reuses WAM's DB and TUI infrastructure) |
| 162 |
2. A new `wam/src/pipeline.rs` view that queries WAM tickets filtered by `source: ci` |
| 163 |
3. A standalone binary if the scope grows beyond CI/CD |
| 164 |
|
| 165 |
--- |
| 166 |
|
| 167 |
## Existing Infrastructure |
| 168 |
|
| 169 |
|
| 170 |
|
| 171 |
| CI runner | astra:/home/max/staging/run-ci.sh | exists | |
| 172 |
| CI trigger | deploy/ci-on-push.sh | written, needs deploy to astra | |
| 173 |
| Post-receive hook | deploy/post-receive-hook.sh | written, needs install | |
| 174 |
| Setup script | deploy/setup-ci.sh | written, needs astra SSH | |
| 175 |
| Test database | astra PostgreSQL | exists | |
| 176 |
| Deploy script | deploy/deploy.sh | exists, manual | |
| 177 |
| PoM monitoring | astra + hetzner (peer mesh) | exists, 5min interval | |
| 178 |
| WAM tickets | hetzner:7890 | running | |
| 179 |
| WAM TUI | wam binary | running, needs pipeline view | |
| 180 |
|
| 181 |
## Astra CI Environment |
| 182 |
|
| 183 |
Required: |
| 184 |
- Rust toolchain (stable) with `x86_64-unknown-linux-gnu` target |
| 185 |
- cargo-zigbuild + zig (for x86_64 cross-compilation) |
| 186 |
- PostgreSQL running (test databases created/dropped per test) |
| 187 |
- SSH access to Hetzner port 2200 (for scp binary upload) |
| 188 |
- `WAM_URL=http://100.120.174.96:7890` in environment or `.bashrc` |
| 189 |
- `SQLX_OFFLINE=true` in environment |
| 190 |
- `TEST_DATABASE_URL=postgres:///postgres` in environment |
| 191 |
|
| 192 |
## File Inventory |
| 193 |
|
| 194 |
``` |
| 195 |
server/deploy/ |
| 196 |
deploy.sh # existing manual deploy (cross-compile + upload + restart) |
| 197 |
run-ci.sh # existing CI script (check, test, clippy, audit) |
| 198 |
ci-on-push.sh # NEW: pull + CI + WAM ticket wrapper |
| 199 |
post-receive-hook.sh # NEW: git hook (CI trigger on main, OTA on tags) |
| 200 |
setup-ci.sh # NEW: one-time provisioning (SSH key, hook install) |
| 201 |
``` |
| 202 |
|