Skip to main content

max / makenotwork

Add CI/CD pipeline scripts and roadmap Push-triggered CI: post-receive hook on Hetzner SSHes to astra on main push, runs full test suite, reports results via WAM ticket. Scripts: - ci-on-push.sh: wrapper that pulls, runs CI, creates WAM ticket - post-receive-hook.sh: git hook (CI on branch push, OTA on tag push) - setup-ci.sh: one-time provisioning (SSH key, hook install) - cicd.md: full 5-step roadmap including staging, promote, and TUI monitor Pending: astra SSH access to complete provisioning. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author: Max J. <87768334+MaxJMath@users.noreply.github.com> · 2026-04-25 20:28 UTC
Commit: cfb7e9e7bdf80bf21f62f55de5c04cd8872757ac
Parent: ca4f6e7
4 files changed, +402 insertions, -0 deletions
A server/deploy/ci-on-push.sh +98
@@ -0,0 +1,98 @@
1 + #!/bin/bash
2 + # CI wrapper for push-triggered builds on astra.
3 + # Called by the post-receive hook on Hetzner via SSH.
4 + #
5 + # Pulls latest code, runs the CI suite, and reports results to WAM.
6 + #
7 + # Location on astra: /home/max/staging/ci-on-push.sh
8 + # Usage: ci-on-push.sh [branch_name]
9 +
10 + set -uo pipefail
11 +
12 + export PATH="$HOME/.cargo/bin:$PATH"
13 +
14 + BRANCH="${1:-main}"
15 + STAGING_DIR="$HOME/staging"
16 + SERVER_DIR="$STAGING_DIR/server"
17 + WAM_URL="${WAM_URL:-http://100.120.174.96:7890}"
18 + START_TIME=$(date +%s)
19 + LOG_FILE="$STAGING_DIR/ci-latest.log"
20 +
21 + echo "=== CI triggered for branch: $BRANCH ==="
22 + echo "Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
23 +
24 + # Pull latest code
25 + cd "$STAGING_DIR" || exit 1
26 +
27 + if [ -d ".git" ]; then
28 + echo "[pull] Fetching latest..."
29 + git fetch origin "$BRANCH" 2>&1
30 + git reset --hard "origin/$BRANCH" 2>&1
31 + else
32 + echo "[pull] No git repo in staging dir, skipping pull"
33 + fi
34 +
35 + # Run CI (capture output for WAM ticket)
36 + cd "$SERVER_DIR" 2>/dev/null || cd "$STAGING_DIR" || exit 1
37 +
38 + echo "[ci] Running CI suite..."
39 + CI_OUTPUT=$("$STAGING_DIR/run-ci.sh" 2>&1) || true
40 + CI_EXIT=$?
41 +
42 + END_TIME=$(date +%s)
43 + DURATION=$(( END_TIME - START_TIME ))
44 +
45 + # Extract summary from CI output
46 + SUMMARY=$(echo "$CI_OUTPUT" | grep -A 50 "CI Summary" | tail -n +2)
47 + PASS_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ PASS" || true)
48 + FAIL_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ FAIL" || true)
49 +
50 + # Save full log
51 + echo "$CI_OUTPUT" > "$LOG_FILE"
52 +
53 + # Determine ticket priority and title
54 + if [ $CI_EXIT -eq 0 ]; then
55 + PRIORITY="low"
56 + TITLE="CI passed: $BRANCH ($PASS_COUNT steps, ${DURATION}s)"
57 + STATUS_LINE="All steps passed."
58 + else
59 + PRIORITY="high"
60 + TITLE="CI failed: $BRANCH ($FAIL_COUNT step(s) failed, ${DURATION}s)"
61 + # Extract failed step names
62 + FAILED_STEPS=$(echo "$CI_OUTPUT" | grep "^ FAIL" | sed 's/^ FAIL / - /')
63 + STATUS_LINE="Failed steps:\n$FAILED_STEPS"
64 + fi
65 +
66 + # Build ticket body (truncate to avoid huge payloads)
67 + BODY=$(cat <<TICKET
68 + Branch: $BRANCH
69 + Duration: ${DURATION}s
70 + Steps passed: $PASS_COUNT
71 + Steps failed: $FAIL_COUNT
72 +
73 + $STATUS_LINE
74 +
75 + Last 30 lines of output:
76 + $(echo "$CI_OUTPUT" | tail -30)
77 + TICKET
78 + )
79 +
80 + # Create WAM ticket
81 + ESCAPED_BODY=$(echo "$BODY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))' 2>/dev/null || echo '""')
82 +
83 + curl -sf -X POST "$WAM_URL/tickets" \
84 + -H "Content-Type: application/json" \
85 + -d "{
86 + \"title\": $(echo "$TITLE" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read().strip()))'),
87 + \"body\": $ESCAPED_BODY,
88 + \"priority\": \"$PRIORITY\",
89 + \"source\": \"ci\",
90 + \"source_ref\": \"$BRANCH\"
91 + }" >/dev/null 2>&1 || echo "[warn] Failed to create WAM ticket"
92 +
93 + echo ""
94 + echo "$SUMMARY"
95 + echo ""
96 + echo "Duration: ${DURATION}s"
97 + echo "WAM ticket: $PRIORITY"
98 + exit $CI_EXIT
A server/deploy/post-receive-hook.sh +37
@@ -0,0 +1,37 @@
1 + #!/bin/bash
2 + # Git post-receive hook for makenotwork.git on Hetzner.
3 + #
4 + # Handles two events:
5 + # 1. Tag push (v*): triggers OTA build via MNW internal API (existing)
6 + # 2. Branch push (main): triggers CI on astra via SSH (new)
7 + #
8 + # Install: copy to /opt/git/max/makenotwork.git/hooks/post-receive
9 +
10 + ASTRA_HOST="max@100.106.221.39"
11 + ASTRA_CI_SCRIPT="/home/max/staging/ci-on-push.sh"
12 + BUILD_TOKEN="a11d2b9ff121a70e7391b36d00e812da6bba1dd91ed17d4b5e0f0dbb7f7f66cc"
13 +
14 + while read oldrev newrev refname; do
15 + case "$refname" in
16 + refs/tags/v[0-9]*)
17 + # OTA build trigger (existing behavior)
18 + TAG="${refname#refs/tags/}"
19 + REPO_PATH="$(cd "$(dirname "$0")/.." && pwd)"
20 + REPO_NAME="$(basename "$REPO_PATH" .git)"
21 + OWNER="$(basename "$(dirname "$REPO_PATH")")"
22 + curl -sf -X POST \
23 + -H "Authorization: Bearer $BUILD_TOKEN" \
24 + -H "Content-Type: application/json" \
25 + -d "{\"repo_owner\": \"$OWNER\", \"repo_name\": \"$REPO_NAME\", \"tag\": \"$TAG\"}" \
26 + "http://localhost:3000/api/internal/builds/trigger" \
27 + >/dev/null 2>&1 &
28 + ;;
29 + refs/heads/main)
30 + # CI trigger: SSH to astra in background
31 + # Uses the git user's SSH key (set up during CI provisioning)
32 + ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
33 + "$ASTRA_HOST" "$ASTRA_CI_SCRIPT main" \
34 + </dev/null >/dev/null 2>&1 &
35 + ;;
36 + esac
37 + done
A server/deploy/setup-ci.sh +66
@@ -0,0 +1,66 @@
1 + #!/bin/bash
2 + # One-time CI setup: generate SSH key for git user on Hetzner,
3 + # install public key on astra, deploy hook and CI script.
4 + #
5 + # Run from MNW/server/ on the MacBook.
6 +
7 + set -e
8 +
9 + HETZNER="root@100.120.174.96"
10 + HETZNER_SSH="-p 2200"
11 + ASTRA="max@100.106.221.39"
12 + HOOK_SRC="deploy/post-receive-hook.sh"
13 + CI_SRC="deploy/ci-on-push.sh"
14 +
15 + echo "=== CI Setup ==="
16 +
17 + # Step 1: Generate SSH key for git user on Hetzner (if not exists)
18 + echo "[1/5] Generating SSH key for git user on Hetzner..."
19 + ssh $HETZNER_SSH $HETZNER bash -s <<'REMOTE'
20 + if [ ! -f /opt/git/.ssh/id_ed25519 ]; then
21 + sudo -u git ssh-keygen -t ed25519 -f /opt/git/.ssh/id_ed25519 -N "" -C "git@hetzner-ci"
22 + chown git:git /opt/git/.ssh/id_ed25519 /opt/git/.ssh/id_ed25519.pub
23 + echo "Key generated."
24 + else
25 + echo "Key already exists."
26 + fi
27 + cat /opt/git/.ssh/id_ed25519.pub
28 + REMOTE
29 +
30 + # Step 2: Get the public key
31 + echo ""
32 + echo "[2/5] Retrieving public key..."
33 + PUBKEY=$(ssh $HETZNER_SSH $HETZNER "cat /opt/git/.ssh/id_ed25519.pub")
34 + echo " $PUBKEY"
35 +
36 + # Step 3: Install public key on astra
37 + echo ""
38 + echo "[3/5] Installing public key on astra..."
39 + echo " Copy this key to astra's ~/.ssh/authorized_keys:"
40 + echo ""
41 + echo " $PUBKEY"
42 + echo ""
43 + echo " Run on astra: echo '$PUBKEY' >> /home/max/.ssh/authorized_keys"
44 + echo ""
45 + read -p " Press Enter once the key is installed on astra..."
46 +
47 + # Step 4: Deploy post-receive hook
48 + echo ""
49 + echo "[4/5] Deploying post-receive hook..."
50 + scp $HETZNER_SSH $HOOK_SRC $HETZNER:/opt/git/max/makenotwork.git/hooks/post-receive
51 + ssh $HETZNER_SSH $HETZNER "chown git:git /opt/git/max/makenotwork.git/hooks/post-receive && chmod +x /opt/git/max/makenotwork.git/hooks/post-receive"
52 + echo " Hook installed."
53 +
54 + # Step 5: Deploy CI script to astra
55 + echo ""
56 + echo "[5/5] Deploying CI script to astra..."
57 + echo " Copy $CI_SRC to astra:/home/max/staging/ci-on-push.sh"
58 + echo " (Cannot scp to astra from here if SSH is down — copy manually)"
59 + echo ""
60 + echo " scp $CI_SRC $ASTRA:/home/max/staging/ci-on-push.sh"
61 + echo " ssh $ASTRA 'chmod +x /home/max/staging/ci-on-push.sh'"
62 +
63 + echo ""
64 + echo "=== Setup Complete ==="
65 + echo ""
66 + echo "Test: push a commit to mnw/main and check WAM for a CI ticket."
A server/docs/cicd.md +201
@@ -0,0 +1,201 @@
1 + # CI/CD Pipeline
2 +
3 + ## Overview
4 +
5 + Push-triggered CI on astra, WAM ticket notifications, staged promotion to Hetzner production. Monitored via a ratatui TUI on the tailnet.
6 +
7 + ## Architecture
8 +
9 + ```
10 + MacBook (dev)
11 + |
12 + | git push mnw main
13 + v
14 + Hetzner (production, x86_64)
15 + |
16 + | post-receive hook: ssh astra
17 + v
18 + Astra (CI/build, aarch64 Linux)
19 + |
20 + | 1. cargo check
21 + | 2. cargo test --lib
22 + | 3. cargo test --test integration
23 + | 4. cargo clippy
24 + | 5. cargo audit
25 + |
26 + | on success:
27 + | - WAM ticket (pass, test count, duration)
28 + | - cross-compile x86_64 binary (cargo-zigbuild)
29 + | - scp binary to Hetzner staging dir
30 + |
31 + | on failure:
32 + | - WAM ticket (fail, which step, error)
33 + | - stop (do not build or promote)
34 + |
35 + v
36 + Hetzner staging (/opt/makenotwork-staging/)
37 + |
38 + | PoM health checks staging instance on port 3001
39 + |
40 + | manual or TUI-driven promote
41 + v
42 + Hetzner production (/opt/makenotwork/)
43 + ```
44 +
45 + ## Machines
46 +
47 + | Machine | Role | Arch | Tailscale IP |
48 + |---------|------|------|-------------|
49 + | MacBook | Development | aarch64 macOS | local |
50 + | Astra | CI runner, native aarch64 + cross-compile x86_64 | aarch64 Linux | 100.106.221.39 |
51 + | Hetzner | Production server, staging host | x86_64 Linux | 100.120.174.96 |
52 +
53 + Astra is aarch64 Linux. It runs tests natively and cross-compiles for x86_64 (Hetzner) via cargo-zigbuild.
54 +
55 + ---
56 +
57 + ## Step 1: Push-triggered CI -- IMPLEMENTED (pending astra SSH)
58 +
59 + **Scripts:**
60 + - `deploy/post-receive-hook.sh` -- git hook for Hetzner (triggers CI on main push, OTA on tag push)
61 + - `deploy/ci-on-push.sh` -- wrapper on astra (pulls, runs CI, reports to WAM)
62 + - `deploy/setup-ci.sh` -- one-time provisioning (SSH key, hook install)
63 +
64 + **Trigger:** Git post-receive hook on Hetzner SSHes to astra on `refs/heads/main` push.
65 +
66 + **What runs:** `/home/max/staging/ci-on-push.sh` calls existing `run-ci.sh`:
67 + - cargo check
68 + - cargo test --lib (unit tests)
69 + - cargo test --test integration (8 threads, isolated per-test databases)
70 + - cargo clippy
71 + - cargo audit
72 +
73 + **Notification:** WAM ticket created with pass/fail, step counts, duration, last 30 lines of output.
74 +
75 + **Setup required:**
76 + 1. Generate SSH key for git user on Hetzner (`deploy/setup-ci.sh` handles this)
77 + 2. Install public key on astra `~/.ssh/authorized_keys`
78 + 3. Copy `ci-on-push.sh` to astra `/home/max/staging/`
79 + 4. Install hook on Hetzner (setup script handles this)
80 + 5. Astra needs: `WAM_URL=http://100.120.174.96:7890` in environment
81 +
82 + ---
83 +
84 + ## Step 2: Auto-build on CI success
85 +
86 + On CI pass, `ci-on-push.sh` continues:
87 + 1. Cross-compile x86_64 release binary on astra (`cargo zigbuild --release --target x86_64-unknown-linux-gnu`)
88 + 2. scp binary to Hetzner `/opt/makenotwork-staging/makenotwork`
89 + 3. WAM ticket: "build ready for staging"
90 +
91 + Astra prerequisites:
92 + - `rustup target add x86_64-unknown-linux-gnu`
93 + - `cargo install cargo-zigbuild` (or system zig + zigbuild)
94 +
95 + ---
96 +
97 + ## Step 3: Staging environment
98 +
99 + Hetzner runs a second instance for validation before production:
100 + - Binary at `/opt/makenotwork-staging/makenotwork`
101 + - Systemd unit: `makenotwork-staging.service` (port 3001)
102 + - Database: `makenotwork_staging` (separate from production)
103 + - Env: copy of production `.env` with different port and DB URL
104 + - PoM health-checks staging at `http://100.120.174.96:3001/api/health`
105 + - WAM ticket on staging health pass: "staging healthy, ready to promote"
106 +
107 + ---
108 +
109 + ## Step 4: Promote to production
110 +
111 + `deploy/promote.sh`:
112 + 1. Send 30s restart warning via internal API
113 + 2. Stop production service
114 + 3. Copy staging binary to `/opt/makenotwork/makenotwork`
115 + 4. Start production service
116 + 5. PoM verifies health within 30s
117 + 6. If unhealthy: rollback (restore previous binary from `/opt/makenotwork/makenotwork.prev`, restart)
118 + 7. WAM ticket: promoted or rolled back
119 +
120 + ---
121 +
122 + ## Step 5: CI/CD Monitor TUI
123 +
124 + A ratatui-based TUI (part of WAM or standalone) for real-time pipeline visibility across the tailnet. Connects to WAM's HTTP API to display CI/CD state.
125 +
126 + ### Views
127 +
128 + **Pipeline dashboard (default):**
129 + ```
130 + CI/CD v0.4.0 -> v0.4.1
131 + ---------------------------------------------------------------
132 + [v] check [v] test-lib [v] test-int [v] clippy [v] audit
133 + [v] build [v] staging [ ] promote
134 + ---------------------------------------------------------------
135 + Last CI: 2m ago (42s, 715 tests passed)
136 + Staging: healthy (3m uptime)
137 + Prod: v0.4.0 (2h uptime)
138 + ```
139 +
140 + **Recent runs:**
141 + - List of recent CI runs with pass/fail, duration, commit hash
142 + - Filter by branch, status
143 + - Enter to view full log
144 +
145 + **Actions (keyboard-driven):**
146 + - `p` -- promote staging to production (with confirmation)
147 + - `r` -- re-run CI manually (SSH to astra)
148 + - `s` -- view staging health details
149 + - `l` -- view full CI log
150 + - `d` -- deploy current staging binary (skip CI)
151 +
152 + **Data sources:**
153 + - WAM tickets (source: `ci`) for CI run results
154 + - WAM tickets (source: `build-failed`, `health-status-change`) for problems
155 + - PoM API for staging + production health status
156 + - SSH to astra for live CI log streaming (if a run is in progress)
157 +
158 + ### Implementation
159 +
160 + Could be:
161 + 1. A new `wam pipeline` subcommand (simplest, reuses WAM's DB and TUI infrastructure)
162 + 2. A new `wam/src/pipeline.rs` view that queries WAM tickets filtered by `source: ci`
163 + 3. A standalone binary if the scope grows beyond CI/CD
164 +
165 + ---
166 +
167 + ## Existing Infrastructure
168 +
169 + | Component | Location | Status |
170 + |-----------|----------|--------|
171 + | CI runner | astra:/home/max/staging/run-ci.sh | exists |
172 + | CI trigger | deploy/ci-on-push.sh | written, needs deploy to astra |
173 + | Post-receive hook | deploy/post-receive-hook.sh | written, needs install |
174 + | Setup script | deploy/setup-ci.sh | written, needs astra SSH |
175 + | Test database | astra PostgreSQL | exists |
176 + | Deploy script | deploy/deploy.sh | exists, manual |
177 + | PoM monitoring | astra + hetzner (peer mesh) | exists, 5min interval |
178 + | WAM tickets | hetzner:7890 | running |
179 + | WAM TUI | wam binary | running, needs pipeline view |
180 +
181 + ## Astra CI Environment
182 +
183 + Required:
184 + - Rust toolchain (stable) with `x86_64-unknown-linux-gnu` target
185 + - cargo-zigbuild + zig (for x86_64 cross-compilation)
186 + - PostgreSQL running (test databases created/dropped per test)
187 + - SSH access to Hetzner port 2200 (for scp binary upload)
188 + - `WAM_URL=http://100.120.174.96:7890` in environment or `.bashrc`
189 + - `SQLX_OFFLINE=true` in environment
190 + - `TEST_DATABASE_URL=postgres:///postgres` in environment
191 +
192 + ## File Inventory
193 +
194 + ```
195 + server/deploy/
196 + deploy.sh # existing manual deploy (cross-compile + upload + restart)
197 + run-ci.sh # existing CI script (check, test, clippy, audit)
198 + ci-on-push.sh # NEW: pull + CI + WAM ticket wrapper
199 + post-receive-hook.sh # NEW: git hook (CI trigger on main, OTA on tags)
200 + setup-ci.sh # NEW: one-time provisioning (SSH key, hook install)
201 + ```