Skip to main content
maxmakenotwork

Add CI/CD pipeline scripts and roadmap Push-triggered CI: post-receive hook on Hetzner SSHes to astra on main push, runs full test suite, reports results via WAM ticket. Scripts: - ci-on-push.sh: wrapper that pulls, runs CI, creates WAM ticket - post-receive-hook.sh: git hook (CI on branch push, OTA on tag push) - setup-ci.sh: one-time provisioning (SSH key, hook install) - cicd.md: full 5-step roadmap including staging, promote, and TUI monitor Pending: astra SSH access to complete provisioning.

  • Co-Authored-ByClaude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: Max J. <87768334+MaxJMath@users.noreply.github.com> - 2026-04-25 20:28 UTC

Commit:

cfb7e9e7bdf80bf21f62f55de5c04cd8872757ac

Parent:

ca4f6e7

4 files changed,

+402 insertions,

-0 deletions

Aserver/deploy/ci-on-push.sh

+98

OldNewLine
@@ -1,0 +1,98 @@
1
#!/bin/bash
2
# CI wrapper for push-triggered builds on astra.
3
# Called by the post-receive hook on Hetzner via SSH.
4
#
5
# Pulls latest code, runs the CI suite, and reports results to WAM.
6
#
7
# Location on astra: /home/max/staging/ci-on-push.sh
8
# Usage: ci-on-push.sh [branch_name]
9
10
set -uo pipefail
11
12
export PATH="$HOME/.cargo/bin:$PATH"
13
14
BRANCH="${1:-main}"
15
STAGING_DIR="$HOME/staging"
16
SERVER_DIR="$STAGING_DIR/server"
17
WAM_URL="${WAM_URL:-http://100.120.174.96:7890}"
18
START_TIME=$(date +%s)
19
LOG_FILE="$STAGING_DIR/ci-latest.log"
20
21
echo "=== CI triggered for branch: $BRANCH ==="
22
echo "Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
23
24
# Pull latest code
25
cd "$STAGING_DIR" || exit 1
26
27
if [ -d ".git" ]; then
28
echo "[pull] Fetching latest..."
29
git fetch origin "$BRANCH" 2>&1
30
git reset --hard "origin/$BRANCH" 2>&1
31
else
32
echo "[pull] No git repo in staging dir, skipping pull"
33
fi
34
35
# Run CI (capture output for WAM ticket)
36
cd "$SERVER_DIR" 2>/dev/null || cd "$STAGING_DIR" || exit 1
37
38
echo "[ci] Running CI suite..."
39
CI_OUTPUT=$("$STAGING_DIR/run-ci.sh" 2>&1) || true
40
CI_EXIT=$?
41
42
END_TIME=$(date +%s)
43
DURATION=$(( END_TIME - START_TIME ))
44
45
# Extract summary from CI output
46
SUMMARY=$(echo "$CI_OUTPUT" | grep -A 50 "CI Summary" | tail -n +2)
47
PASS_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ PASS" || true)
48
FAIL_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ FAIL" || true)
49
50
# Save full log
51
echo "$CI_OUTPUT" > "$LOG_FILE"
52
53
# Determine ticket priority and title
54
if [ $CI_EXIT -eq 0 ]; then
55
PRIORITY="low"
56
TITLE="CI passed: $BRANCH ($PASS_COUNT steps, ${DURATION}s)"
57
STATUS_LINE="All steps passed."
58
else
59
PRIORITY="high"
60
TITLE="CI failed: $BRANCH ($FAIL_COUNT step(s) failed, ${DURATION}s)"
61
# Extract failed step names
62
FAILED_STEPS=$(echo "$CI_OUTPUT" | grep "^ FAIL" | sed 's/^ FAIL / - /')
63
STATUS_LINE="Failed steps:\n$FAILED_STEPS"
64
fi
65
66
# Build ticket body (truncate to avoid huge payloads)
67
BODY=$(cat <<TICKET
68
Branch: $BRANCH
69
Duration: ${DURATION}s
70
Steps passed: $PASS_COUNT
71
Steps failed: $FAIL_COUNT
72
73
$STATUS_LINE
74
75
Last 30 lines of output:
76
$(echo "$CI_OUTPUT" | tail -30)
77
TICKET
78
)
79
80
# Create WAM ticket
81
ESCAPED_BODY=$(echo "$BODY" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))' 2>/dev/null || echo '""')
82
83
curl -sf -X POST "$WAM_URL/tickets" \
84
-H "Content-Type: application/json" \
85
-d "{
86
\"title\": $(echo "$TITLE" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read().strip()))'),
87
\"body\": $ESCAPED_BODY,
88
\"priority\": \"$PRIORITY\",
89
\"source\": \"ci\",
90
\"source_ref\": \"$BRANCH\"
91
}" >/dev/null 2>&1 || echo "[warn] Failed to create WAM ticket"
92
93
echo ""
94
echo "$SUMMARY"
95
echo ""
96
echo "Duration: ${DURATION}s"
97
echo "WAM ticket: $PRIORITY"
98
exit $CI_EXIT
Aserver/deploy/post-receive-hook.sh

+37

OldNewLine
@@ -1,0 +1,37 @@
1
#!/bin/bash
2
# Git post-receive hook for makenotwork.git on Hetzner.
3
#
4
# Handles two events:
5
# 1. Tag push (v*): triggers OTA build via MNW internal API (existing)
6
# 2. Branch push (main): triggers CI on astra via SSH (new)
7
#
8
# Install: copy to /opt/git/max/makenotwork.git/hooks/post-receive
9
10
ASTRA_HOST="max@100.106.221.39"
11
ASTRA_CI_SCRIPT="/home/max/staging/ci-on-push.sh"
12
BUILD_TOKEN="a11d2b9ff121a70e7391b36d00e812da6bba1dd91ed17d4b5e0f0dbb7f7f66cc"
13
14
while read oldrev newrev refname; do
15
case "$refname" in
16
refs/tags/v[0-9]*)
17
# OTA build trigger (existing behavior)
18
TAG="${refname#refs/tags/}"
19
REPO_PATH="$(cd "$(dirname "$0")/.." && pwd)"
20
REPO_NAME="$(basename "$REPO_PATH" .git)"
21
OWNER="$(basename "$(dirname "$REPO_PATH")")"
22
curl -sf -X POST \
23
-H "Authorization: Bearer $BUILD_TOKEN" \
24
-H "Content-Type: application/json" \
25
-d "{\"repo_owner\": \"$OWNER\", \"repo_name\": \"$REPO_NAME\", \"tag\": \"$TAG\"}" \
26
"http://localhost:3000/api/internal/builds/trigger" \
27
>/dev/null 2>&1 &
28
;;
29
refs/heads/main)
30
# CI trigger: SSH to astra in background
31
# Uses the git user's SSH key (set up during CI provisioning)
32
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
33
"$ASTRA_HOST" "$ASTRA_CI_SCRIPT main" \
34
</dev/null >/dev/null 2>&1 &
35
;;
36
esac
37
done
Aserver/deploy/setup-ci.sh

+66

OldNewLine
@@ -1,0 +1,66 @@
1
#!/bin/bash
2
# One-time CI setup: generate SSH key for git user on Hetzner,
3
# install public key on astra, deploy hook and CI script.
4
#
5
# Run from MNW/server/ on the MacBook.
6
7
set -e
8
9
HETZNER="root@100.120.174.96"
10
HETZNER_SSH="-p 2200"
11
ASTRA="max@100.106.221.39"
12
HOOK_SRC="deploy/post-receive-hook.sh"
13
CI_SRC="deploy/ci-on-push.sh"
14
15
echo "=== CI Setup ==="
16
17
# Step 1: Generate SSH key for git user on Hetzner (if not exists)
18
echo "[1/5] Generating SSH key for git user on Hetzner..."
19
ssh $HETZNER_SSH $HETZNER bash -s <<'REMOTE'
20
if [ ! -f /opt/git/.ssh/id_ed25519 ]; then
21
sudo -u git ssh-keygen -t ed25519 -f /opt/git/.ssh/id_ed25519 -N "" -C "git@hetzner-ci"
22
chown git:git /opt/git/.ssh/id_ed25519 /opt/git/.ssh/id_ed25519.pub
23
echo "Key generated."
24
else
25
echo "Key already exists."
26
fi
27
cat /opt/git/.ssh/id_ed25519.pub
28
REMOTE
29
30
# Step 2: Get the public key
31
echo ""
32
echo "[2/5] Retrieving public key..."
33
PUBKEY=$(ssh $HETZNER_SSH $HETZNER "cat /opt/git/.ssh/id_ed25519.pub")
34
echo " $PUBKEY"
35
36
# Step 3: Install public key on astra
37
echo ""
38
echo "[3/5] Installing public key on astra..."
39
echo " Copy this key to astra's ~/.ssh/authorized_keys:"
40
echo ""
41
echo " $PUBKEY"
42
echo ""
43
echo " Run on astra: echo '$PUBKEY' >> /home/max/.ssh/authorized_keys"
44
echo ""
45
read -p " Press Enter once the key is installed on astra..."
46
47
# Step 4: Deploy post-receive hook
48
echo ""
49
echo "[4/5] Deploying post-receive hook..."
50
scp $HETZNER_SSH $HOOK_SRC $HETZNER:/opt/git/max/makenotwork.git/hooks/post-receive
51
ssh $HETZNER_SSH $HETZNER "chown git:git /opt/git/max/makenotwork.git/hooks/post-receive && chmod +x /opt/git/max/makenotwork.git/hooks/post-receive"
52
echo " Hook installed."
53
54
# Step 5: Deploy CI script to astra
55
echo ""
56
echo "[5/5] Deploying CI script to astra..."
57
echo " Copy $CI_SRC to astra:/home/max/staging/ci-on-push.sh"
58
echo " (Cannot scp to astra from here if SSH is down — copy manually)"
59
echo ""
60
echo " scp $CI_SRC $ASTRA:/home/max/staging/ci-on-push.sh"
61
echo " ssh $ASTRA 'chmod +x /home/max/staging/ci-on-push.sh'"
62
63
echo ""
64
echo "=== Setup Complete ==="
65
echo ""
66
echo "Test: push a commit to mnw/main and check WAM for a CI ticket."
Aserver/docs/cicd.md

+201

OldNewLine
@@ -1,0 +1,201 @@
1
# CI/CD Pipeline
2
3
## Overview
4
5
Push-triggered CI on astra, WAM ticket notifications, staged promotion to Hetzner production. Monitored via a ratatui TUI on the tailnet.
6
7
## Architecture
8
9
```
10
MacBook (dev)
11
|
12
| git push mnw main
13
v
14
Hetzner (production, x86_64)
15
|
16
| post-receive hook: ssh astra
17
v
18
Astra (CI/build, aarch64 Linux)
19
|
20
| 1. cargo check
21
| 2. cargo test --lib
22
| 3. cargo test --test integration
23
| 4. cargo clippy
24
| 5. cargo audit
25
|
26
| on success:
27
| - WAM ticket (pass, test count, duration)
28
| - cross-compile x86_64 binary (cargo-zigbuild)
29
| - scp binary to Hetzner staging dir
30
|
31
| on failure:
32
| - WAM ticket (fail, which step, error)
33
| - stop (do not build or promote)
34
|
35
v
36
Hetzner staging (/opt/makenotwork-staging/)
37
|
38
| PoM health checks staging instance on port 3001
39
|
40
| manual or TUI-driven promote
41
v
42
Hetzner production (/opt/makenotwork/)
43
```
44
45
## Machines
46
47
| Machine | Role | Arch | Tailscale IP |
48
|---------|------|------|-------------|
49
| MacBook | Development | aarch64 macOS | local |
50
| Astra | CI runner, native aarch64 + cross-compile x86_64 | aarch64 Linux | 100.106.221.39 |
51
| Hetzner | Production server, staging host | x86_64 Linux | 100.120.174.96 |
52
53
Astra is aarch64 Linux. It runs tests natively and cross-compiles for x86_64 (Hetzner) via cargo-zigbuild.
54
55
---
56
57
## Step 1: Push-triggered CI -- IMPLEMENTED (pending astra SSH)
58
59
**Scripts:**
60
- `deploy/post-receive-hook.sh` -- git hook for Hetzner (triggers CI on main push, OTA on tag push)
61
- `deploy/ci-on-push.sh` -- wrapper on astra (pulls, runs CI, reports to WAM)
62
- `deploy/setup-ci.sh` -- one-time provisioning (SSH key, hook install)
63
64
**Trigger:** Git post-receive hook on Hetzner SSHes to astra on `refs/heads/main` push.
65
66
**What runs:** `/home/max/staging/ci-on-push.sh` calls existing `run-ci.sh`:
67
- cargo check
68
- cargo test --lib (unit tests)
69
- cargo test --test integration (8 threads, isolated per-test databases)
70
- cargo clippy
71
- cargo audit
72
73
**Notification:** WAM ticket created with pass/fail, step counts, duration, last 30 lines of output.
74
75
**Setup required:**
76
1. Generate SSH key for git user on Hetzner (`deploy/setup-ci.sh` handles this)
77
2. Install public key on astra `~/.ssh/authorized_keys`
78
3. Copy `ci-on-push.sh` to astra `/home/max/staging/`
79
4. Install hook on Hetzner (setup script handles this)
80
5. Astra needs: `WAM_URL=http://100.120.174.96:7890` in environment
81
82
---
83
84
## Step 2: Auto-build on CI success
85
86
On CI pass, `ci-on-push.sh` continues:
87
1. Cross-compile x86_64 release binary on astra (`cargo zigbuild --release --target x86_64-unknown-linux-gnu`)
88
2. scp binary to Hetzner `/opt/makenotwork-staging/makenotwork`
89
3. WAM ticket: "build ready for staging"
90
91
Astra prerequisites:
92
- `rustup target add x86_64-unknown-linux-gnu`
93
- `cargo install cargo-zigbuild` (or system zig + zigbuild)
94
95
---
96
97
## Step 3: Staging environment
98
99
Hetzner runs a second instance for validation before production:
100
- Binary at `/opt/makenotwork-staging/makenotwork`
101
- Systemd unit: `makenotwork-staging.service` (port 3001)
102
- Database: `makenotwork_staging` (separate from production)
103
- Env: copy of production `.env` with different port and DB URL
104
- PoM health-checks staging at `http://100.120.174.96:3001/api/health`
105
- WAM ticket on staging health pass: "staging healthy, ready to promote"
106
107
---
108
109
## Step 4: Promote to production
110
111
`deploy/promote.sh`:
112
1. Send 30s restart warning via internal API
113
2. Stop production service
114
3. Copy staging binary to `/opt/makenotwork/makenotwork`
115
4. Start production service
116
5. PoM verifies health within 30s
117
6. If unhealthy: rollback (restore previous binary from `/opt/makenotwork/makenotwork.prev`, restart)
118
7. WAM ticket: promoted or rolled back
119
120
---
121
122
## Step 5: CI/CD Monitor TUI
123
124
A ratatui-based TUI (part of WAM or standalone) for real-time pipeline visibility across the tailnet. Connects to WAM's HTTP API to display CI/CD state.
125
126
### Views
127
128
**Pipeline dashboard (default):**
129
```
130
CI/CD v0.4.0 -> v0.4.1
131
---------------------------------------------------------------
132
[v] check [v] test-lib [v] test-int [v] clippy [v] audit
133
[v] build [v] staging [ ] promote
134
---------------------------------------------------------------
135
Last CI: 2m ago (42s, 715 tests passed)
136
Staging: healthy (3m uptime)
137
Prod: v0.4.0 (2h uptime)
138
```
139
140
**Recent runs:**
141
- List of recent CI runs with pass/fail, duration, commit hash
142
- Filter by branch, status
143
- Enter to view full log
144
145
**Actions (keyboard-driven):**
146
- `p` -- promote staging to production (with confirmation)
147
- `r` -- re-run CI manually (SSH to astra)
148
- `s` -- view staging health details
149
- `l` -- view full CI log
150
- `d` -- deploy current staging binary (skip CI)
151
152
**Data sources:**
153
- WAM tickets (source: `ci`) for CI run results
154
- WAM tickets (source: `build-failed`, `health-status-change`) for problems
155
- PoM API for staging + production health status
156
- SSH to astra for live CI log streaming (if a run is in progress)
157
158
### Implementation
159
160
Could be:
161
1. A new `wam pipeline` subcommand (simplest, reuses WAM's DB and TUI infrastructure)
162
2. A new `wam/src/pipeline.rs` view that queries WAM tickets filtered by `source: ci`
163
3. A standalone binary if the scope grows beyond CI/CD
164
165
---
166
167
## Existing Infrastructure
168
169
| Component | Location | Status |
170
|-----------|----------|--------|
171
| CI runner | astra:/home/max/staging/run-ci.sh | exists |
172
| CI trigger | deploy/ci-on-push.sh | written, needs deploy to astra |
173
| Post-receive hook | deploy/post-receive-hook.sh | written, needs install |
174
| Setup script | deploy/setup-ci.sh | written, needs astra SSH |
175
| Test database | astra PostgreSQL | exists |
176
| Deploy script | deploy/deploy.sh | exists, manual |
177
| PoM monitoring | astra + hetzner (peer mesh) | exists, 5min interval |
178
| WAM tickets | hetzner:7890 | running |
179
| WAM TUI | wam binary | running, needs pipeline view |
180
181
## Astra CI Environment
182
183
Required:
184
- Rust toolchain (stable) with `x86_64-unknown-linux-gnu` target
185
- cargo-zigbuild + zig (for x86_64 cross-compilation)
186
- PostgreSQL running (test databases created/dropped per test)
187
- SSH access to Hetzner port 2200 (for scp binary upload)
188
- `WAM_URL=http://100.120.174.96:7890` in environment or `.bashrc`
189
- `SQLX_OFFLINE=true` in environment
190
- `TEST_DATABASE_URL=postgres:///postgres` in environment
191
192
## File Inventory
193
194
```
195
server/deploy/
196
deploy.sh # existing manual deploy (cross-compile + upload + restart)
197
run-ci.sh # existing CI script (check, test, clippy, audit)
198
ci-on-push.sh # NEW: pull + CI + WAM ticket wrapper
199
post-receive-hook.sh # NEW: git hook (CI trigger on main, OTA on tags)
200
setup-ci.sh # NEW: one-time provisioning (SSH key, hook install)
201
```