# CI/CD Pipeline

## Overview

Push-triggered CI on astra, WAM ticket notifications, staged promotion to Hetzner production. Monitored via a ratatui TUI on the tailnet.

## Architecture

```
MacBook (dev)
  |
  | git push mnw main
  v
Hetzner (production, x86_64)
  |
  | post-receive hook: ssh astra
  v
Astra (CI/build, aarch64 Linux)
  |
  | 1. cargo check
  | 2. cargo test --lib
  | 3. cargo test --test integration
  | 4. cargo clippy
  | 5. cargo audit
  |
  | on success:
  |   - WAM ticket (pass, test count, duration)
  |   - cross-compile x86_64 binary (cargo-zigbuild)
  |   - scp binary to Hetzner staging dir
  |
  | on failure:
  |   - WAM ticket (fail, which step, error)
  |   - stop (do not build or promote)
  |
  v
Hetzner staging (/opt/makenotwork-staging/)
  |
  | PoM health checks staging instance on port 3001
  |
  | manual or TUI-driven promote
  v
Hetzner production (/opt/makenotwork/)
```

## Machines

| Machine | Role | Arch | Tailscale IP |
|---------|------|------|-------------|
| MacBook | Development | aarch64 macOS | local |
| Astra | CI runner, native aarch64 + cross-compile x86_64 | aarch64 Linux | 100.106.221.39 |
| Hetzner | Production server, staging host | x86_64 Linux | 100.120.174.96 |

Astra is aarch64 Linux. It runs tests natively and cross-compiles for x86_64 (Hetzner) via cargo-zigbuild.

---

## Step 1: Push-triggered CI -- IMPLEMENTED (pending astra SSH)

**Scripts:**
- `deploy/post-receive-hook.sh` -- git hook for Hetzner (triggers CI on main push, OTA on tag push)
- `deploy/ci-on-push.sh` -- wrapper on astra (pulls, runs CI, reports to WAM)
- `deploy/setup-ci.sh` -- one-time provisioning (SSH key, hook install)

**Trigger:** Git post-receive hook on Hetzner SSHes to astra on `refs/heads/main` push.

**What runs:** `/home/max/staging/ci-on-push.sh` calls existing `run-ci.sh`:
- cargo check
- cargo test --lib (unit tests)
- cargo test --test integration (8 threads, isolated per-test databases)
- cargo clippy
- cargo audit

**Notification:** WAM ticket created with pass/fail, step counts, duration, last 30 lines of output.

**Setup required:**
1. Generate SSH key for git user on Hetzner (`deploy/setup-ci.sh` handles this)
2. Install public key on astra `~/.ssh/authorized_keys`
3. Copy `ci-on-push.sh` to astra `/home/max/staging/`
4. Install hook on Hetzner (setup script handles this)
5. Astra needs: `WAM_URL=http://100.120.174.96:7890` in environment

---

## Step 2: Auto-build on CI success

On CI pass, `ci-on-push.sh` continues:
1. Cross-compile x86_64 release binary on astra (`cargo zigbuild --release --target x86_64-unknown-linux-gnu`)
2. scp binary to Hetzner `/opt/makenotwork-staging/makenotwork`
3. WAM ticket: "build ready for staging"

Astra prerequisites:
- `rustup target add x86_64-unknown-linux-gnu`
- `cargo install cargo-zigbuild` (or system zig + zigbuild)

---

## Step 3: Staging environment

Hetzner runs a second instance for validation before production:
- Binary at `/opt/makenotwork-staging/makenotwork`
- Systemd unit: `makenotwork-staging.service` (port 3001)
- Database: `makenotwork_staging` (separate from production)
- Env: copy of production `.env` with different port and DB URL
- PoM health-checks staging at `http://100.120.174.96:3001/api/health`
- WAM ticket on staging health pass: "staging healthy, ready to promote"

---

## Step 4: Promote to production

`deploy/promote.sh`:
1. Send 30s restart warning via internal API
2. Stop production service
3. Copy staging binary to `/opt/makenotwork/makenotwork`
4. Start production service
5. PoM verifies health within 30s
6. If unhealthy: rollback (restore previous binary from `/opt/makenotwork/makenotwork.prev`, restart)
7. WAM ticket: promoted or rolled back

---

## Step 5: CI/CD Monitor TUI

A ratatui-based TUI (part of WAM or standalone) for real-time pipeline visibility across the tailnet. Connects to WAM's HTTP API to display CI/CD state.

### Views

**Pipeline dashboard (default):**
```
 CI/CD                                          v0.4.0 -> v0.4.1
 ---------------------------------------------------------------
 [v] check    [v] test-lib   [v] test-int   [v] clippy   [v] audit
 [v] build    [v] staging    [ ] promote
 ---------------------------------------------------------------
 Last CI:  2m ago (42s, 715 tests passed)
 Staging:  healthy (3m uptime)
 Prod:     v0.4.0 (2h uptime)
```

**Recent runs:**
- List of recent CI runs with pass/fail, duration, commit hash
- Filter by branch, status
- Enter to view full log

**Actions (keyboard-driven):**
- `p` -- promote staging to production (with confirmation)
- `r` -- re-run CI manually (SSH to astra)
- `s` -- view staging health details
- `l` -- view full CI log
- `d` -- deploy current staging binary (skip CI)

**Data sources:**
- WAM tickets (source: `ci`) for CI run results
- WAM tickets (source: `build-failed`, `health-status-change`) for problems
- PoM API for staging + production health status
- SSH to astra for live CI log streaming (if a run is in progress)

### Implementation

Could be:
1. A new `wam pipeline` subcommand (simplest, reuses WAM's DB and TUI infrastructure)
2. A new `wam/src/pipeline.rs` view that queries WAM tickets filtered by `source: ci`
3. A standalone binary if the scope grows beyond CI/CD

---

## Existing Infrastructure

| Component | Location | Status |
|-----------|----------|--------|
| CI runner | astra:/home/max/staging/run-ci.sh | exists |
| CI trigger | deploy/ci-on-push.sh | written, needs deploy to astra |
| Post-receive hook | deploy/post-receive-hook.sh | written, needs install |
| Setup script | deploy/setup-ci.sh | written, needs astra SSH |
| Test database | astra PostgreSQL | exists |
| Deploy script | deploy/deploy.sh | exists, manual |
| PoM monitoring | astra + hetzner (peer mesh) | exists, 5min interval |
| WAM tickets | hetzner:7890 | running |
| WAM TUI | wam binary | running, needs pipeline view |

## Astra CI Environment

Required:
- Rust toolchain (stable) with `x86_64-unknown-linux-gnu` target
- cargo-zigbuild + zig (for x86_64 cross-compilation)
- PostgreSQL running (test databases created/dropped per test)
- SSH access to Hetzner port 2200 (for scp binary upload)
- `WAM_URL=http://100.120.174.96:7890` in environment or `.bashrc`
- `SQLX_OFFLINE=true` in environment
- `TEST_DATABASE_URL=postgres:///postgres` in environment

## File Inventory

```
server/deploy/
  deploy.sh              # existing manual deploy (cross-compile + upload + restart)
  run-ci.sh              # existing CI script (check, test, clippy, audit)
  ci-on-push.sh          # NEW: pull + CI + WAM ticket wrapper
  post-receive-hook.sh   # NEW: git hook (CI trigger on main, OTA on tags)
  setup-ci.sh            # NEW: one-time provisioning (SSH key, hook install)
```