Skip to main content

max / makenotwork

Retire the astra CI pipeline Sando's host-tier gates are the real gate now. The astra pipeline had not run since 2026-05-09 (its ~/mnw-ci origin still pointed at the retired mnw-git path), and it could not have reported a failure if it had: ci-on-push.sh:50 ran CI_OUTPUT=$(...) || true and then read CI_EXIT=$?, so CI_EXIT was structurally always 0. Its last run filed a WAM ticket reading "All steps passed" for a log ending "3 step(s) failed". The coverage it nominally held (clippy, fmt --check, cargo audit, cargo deny) landed as Sando gate kinds in a64ec630, so nothing is lost. Removed here: the five deploy scripts and docs/cicd.md, which described a system that no longer exists. Removed on astra: the ci-watcher systemd unit (still active and polling WAM to no effect), ~/mnw-ci (72 GB), and 245 orphaned mnw_test_* databases (3005 MB). The Hetzner astra-CI trigger is gone as well: the generated post-receive template carries no WAM call, so reinstalling hooks removed it from all 18 repos. post-receive-hook.sh:10 carried the live BUILD_TRIGGER_TOKEN in plaintext, committed since cfb7e9e7 and mirrored to three remotes. Since 90f5bee5 the server verifies HMAC(token, "owner:repo"), so that value was the global key, not a per-repo one: a holder could derive a valid trigger for any repo on the server. It has been rotated on alpha-west-1 and every repo hook regenerated against the new token. Deleting the file does not remediate a leak, so the rotation came first. Also drops PoM's [targets.mnw.tests] check from all three configs. It ran /home/max/staging/run-ci.sh over SSH on astra, a path that does not exist and has not for longer than the pipeline was dead. Sando runs the suite before any deploy, so a separate continuous test-runner is redundant. The ssh/parse capability stays in the code, unconfigured.
Author: Max Johnson <me@maxj.phd> · 2026-07-21 22:29 UTC
Commit: de11716571d03595f57dcde919525964010eb467
Parent: 37f8bf5
12 files changed, +4 insertions, -666 deletions
M pom/deploy/pom-astra.toml -6
@@ -48,12 +48,6 @@ spike_threshold = 2.0
48 48 [targets.mnw.tls]
49 49 host = "makenot.work"
50 50
51 - [targets.mnw.tests]
52 - ssh = "max@100.106.221.39"
53 - command = "/home/max/staging/run-ci.sh"
54 - timeout_secs = 600
55 - staleness_days = 7
56 -
57 51 [targets.mt]
58 52 label = "Multithreaded Forum"
59 53 expected_routes = ["/"]
M pom/deploy/pom-hetzner.toml -6
@@ -53,12 +53,6 @@ spike_threshold = 2.0
53 53 [targets.mnw.tls]
54 54 host = "makenot.work"
55 55
56 - [targets.mnw.tests]
57 - ssh = "max@100.106.221.39"
58 - command = "/home/max/staging/run-ci.sh"
59 - timeout_secs = 600
60 - staleness_days = 7
61 -
62 56 [targets.mnw.backups]
63 57 directory = "/opt/makenotwork/backups"
64 58 databases = ["makenotwork"]
M pom/pom.toml -5
@@ -9,8 +9,3 @@ label = "Makenotwork Production"
9 9 [targets.mnw.health]
10 10 url = "https://makenot.work/api/health"
11 11 timeout_secs = 10
12 -
13 - [targets.mnw.tests]
14 - ssh = "max@100.106.221.39"
15 - command = "/home/max/staging/run-ci.sh"
16 - timeout_secs = 600
M sando/README.md +1 -1
@@ -178,7 +178,7 @@ permanently and uninformatively red, which teaches everyone to ignore it.
178 178 Dropping the config file into a crate is what opts it in.
179 179
180 180 Before these existed, `-D warnings` was enforced in exactly one place
181 - (`server/deploy/run-ci.sh`, which dies with the astra pipeline) and
181 + (`server/deploy/run-ci.sh`, which died with the astra pipeline) and
182 182 `cargo fmt --check` nowhere at all.
183 183
184 184 ### Why `hardening_test` exists
M sando/daemon/src/gates.rs +2 -2
@@ -400,7 +400,7 @@ async fn append_to_log(path: &std::path::Path, bytes: &[u8]) {
400 400 /// `test_target`.
401 401 ///
402 402 /// Before this gate, `-D warnings` was enforced in exactly one place —
403 - /// `server/deploy/run-ci.sh`, which dies with the astra pipeline — so lint drift
403 + /// `server/deploy/run-ci.sh`, which died with the astra pipeline — so lint drift
404 404 /// reached prod ungated. Two crates even carried comments referring to "the
405 405 /// -D warnings CI gate" for a gate that did not exist.
406 406 async fn clippy(ctx: &GateCtx, run_id: GateRunId) -> Result<GateOutcome> {
@@ -745,7 +745,7 @@ fn tests_run(stdout: &[u8]) -> u32 {
745 745 /// pre-gate compile and the full test run so both go through one env setup,
746 746 /// and by both test gates so they can't drift apart on env.
747 747 ///
748 - /// `cargo_test` passes `fast-tests`, matching CI (`server/deploy/run-ci.sh`):
748 + /// `cargo_test` passes `fast-tests`, matching what the retired astra CI did:
749 749 /// it relaxes the auth rate-limit burst (5 → 20) and argon2 cost so
750 750 /// signup-heavy + lockout workflow tests complete without hitting Governor
751 751 /// before the hand-rolled lockout check (documented at
M sando/sando.toml +1 -1
@@ -48,7 +48,7 @@ canary = "sequential"
48 48 # coverage.
49 49 #
50 50 # The lint + supply-chain gates. `-D warnings` used to be enforced in exactly
51 - # one place (server/deploy/run-ci.sh, which dies with the astra pipeline) and
51 + # one place (server/deploy/run-ci.sh, which died with the astra pipeline) and
52 52 # `cargo fmt --check` nowhere at all. fmt runs first: it needs no compilation,
53 53 # so a formatting red comes back in seconds instead of after a full build.
54 54 # cargo_audit and cargo_deny only run in crates carrying a triaged
D server/deploy/ci-on-push.sh -147
@@ -1,147 +0,0 @@
1 - #!/bin/bash
2 - # CI wrapper for push-triggered builds on astra.
3 - # Called by the post-receive hook on Hetzner via SSH.
4 - #
5 - # Pulls latest code, runs the CI suite, reports results to WAM.
6 - # If tests fail on main, auto-reverts the offending commit(s) and pushes
7 - # the revert — enforcing the no-regressions rule from operations.md.
8 - #
9 - # Location on astra: /home/max/mnw-ci/server/deploy/ci-on-push.sh
10 - # Usage: ci-on-push.sh [branch_name]
11 -
12 - set -uo pipefail
13 -
14 - export PATH="$HOME/.cargo/bin:$PATH"
15 -
16 - BRANCH="${1:-main}"
17 - REPO_DIR="$HOME/mnw-ci"
18 - SERVER_DIR="$REPO_DIR/server"
19 - WAM_URL="${WAM_URL:-http://127.0.0.1:7890}"
20 - START_TIME=$(date +%s)
21 - LOG_FILE="$REPO_DIR/ci-latest.log"
22 -
23 - echo "=== CI triggered for branch: $BRANCH ==="
24 - echo "Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
25 -
26 - # Pull latest code
27 - cd "$REPO_DIR" || exit 1
28 - echo "[pull] Fetching latest..."
29 - git fetch origin "$BRANCH" 2>&1
30 -
31 - # Record what we're about to test
32 - OLD_HEAD=$(git rev-parse HEAD 2>/dev/null || echo "none")
33 - git reset --hard "origin/$BRANCH" 2>&1
34 - NEW_HEAD=$(git rev-parse HEAD)
35 - COMMIT_MSG=$(git log --oneline -1)
36 -
37 - echo "[pull] $OLD_HEAD -> $NEW_HEAD"
38 - echo "[pull] $COMMIT_MSG"
39 -
40 - # Run CI from the server directory
41 - cd "$SERVER_DIR" || exit 1
42 -
43 - export DATABASE_URL="${DATABASE_URL:-postgres:///makenotwork_staging}"
44 - export TEST_DATABASE_URL="${TEST_DATABASE_URL:-postgres:///postgres}"
45 - export RUST_TEST_THREADS="${RUST_TEST_THREADS:-8}"
46 - export CARGO_INCREMENTAL=0
47 - export RUST_BACKTRACE=1
48 -
49 - echo "[ci] Running CI suite..."
50 - CI_OUTPUT=$("$REPO_DIR/server/deploy/run-ci.sh" 2>&1) || true
51 - CI_EXIT=$?
52 -
53 - END_TIME=$(date +%s)
54 - DURATION=$(( END_TIME - START_TIME ))
55 -
56 - # Extract summary from CI output
57 - PASS_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ PASS" || true)
58 - FAIL_COUNT=$(echo "$CI_OUTPUT" | grep -c "^ FAIL" || true)
59 -
60 - # Save full log
61 - echo "$CI_OUTPUT" > "$LOG_FILE"
62 -
63 - # --- Handle results ---
64 -
65 - wam_ticket() {
66 - local title="$1" body="$2" priority="$3" source_ref="$4"
67 - local escaped_title escaped_body
68 - escaped_title=$(echo "$title" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read().strip()))' 2>/dev/null || echo "\"$title\"")
69 - escaped_body=$(echo "$body" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))' 2>/dev/null || echo '""')
70 - curl -sf -X POST "$WAM_URL/tickets" \
71 - -H "Content-Type: application/json" \
72 - -d "{\"title\": $escaped_title, \"body\": $escaped_body, \"priority\": \"$priority\", \"source\": \"ci\", \"source_ref\": \"$source_ref\"}" \
73 - >/dev/null 2>&1 || echo "[warn] Failed to create WAM ticket"
74 - }
75 -
76 - if [ $CI_EXIT -eq 0 ]; then
77 - # --- CI PASSED ---
78 - TITLE="CI passed: $BRANCH ($PASS_COUNT steps, ${DURATION}s)"
79 - BODY="Commit: $COMMIT_MSG
80 - Duration: ${DURATION}s
81 - Steps passed: $PASS_COUNT
82 -
83 - All steps passed."
84 -
85 - wam_ticket "$TITLE" "$BODY" "low" "$NEW_HEAD"
86 - echo ""
87 - echo "=== CI PASSED ($PASS_COUNT steps, ${DURATION}s) ==="
88 - else
89 - # --- CI FAILED — AUTO-REVERT ---
90 - FAILED_STEPS=$(echo "$CI_OUTPUT" | grep "^ FAIL" | sed 's/^ FAIL / - /')
91 -
92 - echo ""
93 - echo "=== CI FAILED — REVERTING ==="
94 - echo ""
95 -
96 - cd "$REPO_DIR" || exit 1
97 -
98 - # Count how many new commits since old head
99 - if [ "$OLD_HEAD" = "none" ] || [ "$OLD_HEAD" = "$NEW_HEAD" ]; then
100 - # Can't determine what to revert (first run or no change)
101 - REVERT_STATUS="could not determine commits to revert"
102 - else
103 - NEW_COMMITS=$(git rev-list "$OLD_HEAD..$NEW_HEAD" --count 2>/dev/null || echo "0")
104 -
105 - if [ "$NEW_COMMITS" -eq 1 ]; then
106 - # Single commit — revert it
107 - git revert --no-edit HEAD 2>&1
108 - REVERT_STATUS="reverted 1 commit ($NEW_HEAD)"
109 - elif [ "$NEW_COMMITS" -gt 1 ]; then
110 - # Multiple commits — revert the range
111 - git revert --no-edit "$OLD_HEAD..$NEW_HEAD" 2>&1
112 - REVERT_STATUS="reverted $NEW_COMMITS commits ($OLD_HEAD..$NEW_HEAD)"
113 - else
114 - REVERT_STATUS="no new commits to revert"
115 - fi
116 -
117 - # Push the revert back to origin
118 - if git push origin "$BRANCH" 2>&1; then
119 - REVERT_STATUS="$REVERT_STATUS — pushed to origin"
120 - else
121 - REVERT_STATUS="$REVERT_STATUS — PUSH FAILED (manual intervention needed)"
122 - fi
123 - fi
124 -
125 - TITLE="CI FAILED + REVERTED: $BRANCH ($FAIL_COUNT step(s) failed)"
126 - BODY="Commit: $COMMIT_MSG
127 - Duration: ${DURATION}s
128 - Steps passed: $PASS_COUNT
129 - Steps failed: $FAIL_COUNT
130 -
131 - Failed steps:
132 - $FAILED_STEPS
133 -
134 - Revert: $REVERT_STATUS
135 -
136 - No-regressions rule enforced automatically.
137 - Fix the issue and re-push.
138 -
139 - Last 30 lines of output:
140 - $(echo "$CI_OUTPUT" | tail -30)"
141 -
142 - wam_ticket "$TITLE" "$BODY" "critical" "$NEW_HEAD"
143 - echo "Revert: $REVERT_STATUS"
144 - echo ""
145 - fi
146 -
147 - exit $CI_EXIT
D server/deploy/ci-watcher.sh -62
@@ -1,62 +0,0 @@
1 - #!/bin/bash
2 - # CI watcher for astra — polls WAM for ci-trigger tickets and runs CI.
3 - #
4 - # Runs as a background service on astra. Checks WAM every 30 seconds for
5 - # open tickets with source=ci-trigger. When found, claims the ticket
6 - # (marks it in-progress), runs CI, and updates the ticket with results.
7 - #
8 - # Location on astra: /home/max/mnw-ci/server/deploy/ci-watcher.sh
9 - # Systemd unit: ci-watcher.service
10 - #
11 - # Usage: ci-watcher.sh
12 -
13 - set -uo pipefail
14 -
15 - export PATH="$HOME/.cargo/bin:$PATH"
16 -
17 - WAM_URL="${WAM_URL:-http://100.120.174.96:7890}"
18 - REPO_DIR="$HOME/mnw-ci"
19 - POLL_INTERVAL=30
20 - CI_SCRIPT="$REPO_DIR/server/deploy/ci-on-push.sh"
21 -
22 - echo "CI watcher started (polling WAM every ${POLL_INTERVAL}s)"
23 -
24 - while true; do
25 - sleep "$POLL_INTERVAL"
26 -
27 - # Check for open ci-trigger tickets
28 - RESPONSE=$(curl -sf "$WAM_URL/tickets?source=ci-trigger&status=open" 2>/dev/null) || continue
29 - COUNT=$(echo "$RESPONSE" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("count",0))' 2>/dev/null || echo "0")
30 -
31 - if [ "$COUNT" = "0" ] || [ -z "$COUNT" ]; then
32 - continue
33 - fi
34 -
35 - # Get the first trigger ticket
36 - TICKET_ID=$(echo "$RESPONSE" | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"][0]["id"])' 2>/dev/null) || continue
37 - COMMIT=$(echo "$RESPONSE" | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"][0].get("source_ref","unknown"))' 2>/dev/null || echo "unknown")
38 -
39 - echo ""
40 - echo "=== CI trigger found: $COMMIT ==="
41 -
42 - # Claim the ticket (mark in-progress so we don't pick it up again)
43 - curl -sf -X PATCH "$WAM_URL/tickets/$TICKET_ID" \
44 - -H "Content-Type: application/json" \
45 - -d '{"status": "in-progress"}' >/dev/null 2>&1
46 -
47 - # Run CI
48 - if "$CI_SCRIPT" main 2>&1; then
49 - # CI passed — close the trigger ticket
50 - curl -sf -X PATCH "$WAM_URL/tickets/$TICKET_ID" \
51 - -H "Content-Type: application/json" \
52 - -d '{"status": "closed"}' >/dev/null 2>&1
53 - else
54 - # CI failed — ci-on-push.sh already created a failure ticket and reverted.
55 - # Close the trigger ticket.
56 - curl -sf -X PATCH "$WAM_URL/tickets/$TICKET_ID" \
57 - -H "Content-Type: application/json" \
58 - -d '{"status": "closed"}' >/dev/null 2>&1
59 - fi
60 -
61 - echo "=== CI run complete ==="
62 - done
D server/deploy/post-receive-hook.sh -38
@@ -1,38 +0,0 @@
1 - #!/bin/bash
2 - # Git post-receive hook for makenotwork.git on Hetzner.
3 - #
4 - # Handles two events:
5 - # 1. Tag push (v*): triggers OTA build via MNW internal API (existing)
6 - # 2. Branch push (main): triggers CI on astra via SSH (new)
7 - #
8 - # Install: copy to /opt/git/max/makenotwork.git/hooks/post-receive
9 -
10 - BUILD_TOKEN="a11d2b9ff121a70e7391b36d00e812da6bba1dd91ed17d4b5e0f0dbb7f7f66cc"
11 - # Astra's WAM (Hetzner can reach astra, but not vice versa due to Tailscale ACLs)
12 - ASTRA_WAM_URL="http://100.106.221.39:7890"
13 -
14 - while read oldrev newrev refname; do
15 - case "$refname" in
16 - refs/tags/v[0-9]*)
17 - # OTA build trigger (existing behavior)
18 - TAG="${refname#refs/tags/}"
19 - REPO_PATH="$(cd "$(dirname "$0")/.." && pwd)"
20 - REPO_NAME="$(basename "$REPO_PATH" .git)"
21 - OWNER="$(basename "$(dirname "$REPO_PATH")")"
22 - curl -sf -X POST \
23 - -H "Authorization: Bearer $BUILD_TOKEN" \
24 - -H "Content-Type: application/json" \
25 - -d "{\"repo_owner\": \"$OWNER\", \"repo_name\": \"$REPO_NAME\", \"tag\": \"$TAG\"}" \
26 - "http://localhost:3000/api/internal/builds/trigger" \
27 - >/dev/null 2>&1 &
28 - ;;
29 - refs/heads/main)
30 - # CI trigger: create a WAM ticket that astra's CI watcher picks up.
31 - # Avoids SSH ACL issues — both machines can reach WAM on localhost/tailnet.
32 - curl -sf -X POST "$ASTRA_WAM_URL/tickets" \
33 - -H "Content-Type: application/json" \
34 - -d "{\"title\": \"CI trigger: main pushed ($newrev)\", \"priority\": \"medium\", \"source\": \"ci-trigger\", \"source_ref\": \"$newrev\"}" \
35 - >/dev/null 2>&1 &
36 - ;;
37 - esac
38 - done
D server/deploy/run-ci.sh -131
@@ -1,131 +0,0 @@
1 - #!/bin/bash
2 - # CI script for makenotwork — replaces .build.yml (Sourcehut).
3 - # Run on astra: /home/max/staging/run-ci.sh [filter]
4 - #
5 - # Runs: cargo check, cargo test --lib, cargo test --test integration,
6 - # cargo clippy, cargo audit (if installed).
7 - #
8 - # Usage:
9 - # ./run-ci.sh # full CI
10 - # ./run-ci.sh auth # only tests matching "auth"
11 -
12 - set -euo pipefail
13 -
14 - # Ensure ~/.cargo/bin is in PATH (SSH non-login shells may not source .profile)
15 - export PATH="$HOME/.cargo/bin:$PATH"
16 -
17 - SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
18 - PROJECT_DIR="${SCRIPT_DIR}"
19 -
20 - # Detect project directory layout
21 - if [ -d "$SCRIPT_DIR/makenotwork/src" ]; then
22 - # Rsync'd staging layout (e.g., ~/staging/makenotwork/)
23 - PROJECT_DIR="$SCRIPT_DIR/makenotwork"
24 - fi
25 -
26 - cd "$PROJECT_DIR"
27 -
28 - export DATABASE_URL="${DATABASE_URL:-postgres:///makenotwork_staging}"
29 - export TEST_DATABASE_URL="${TEST_DATABASE_URL:-postgres:///postgres}"
30 - export RUST_TEST_THREADS="${RUST_TEST_THREADS:-8}"
31 - export CARGO_INCREMENTAL=0
32 - export RUST_BACKTRACE=1
33 -
34 - FILTER="${1:-}"
35 - FAILURES=()
36 - PASS=()
37 -
38 - run_step() {
39 - local name="$1"
40 - shift
41 - echo ""
42 - echo "========================================"
43 - echo " $name"
44 - echo "========================================"
45 - echo ""
46 - if "$@"; then
47 - PASS+=("$name")
48 - else
49 - FAILURES+=("$name")
50 - echo "FAILED: $name"
51 - fi
52 - }
53 -
54 - cleanup_test_dbs() {
55 - echo "Cleaning up orphaned test databases..."
56 - sudo -u postgres psql -Atc "SELECT datname FROM pg_database WHERE datname LIKE 'test_%';" 2>/dev/null | while read -r db; do
57 - sudo -u postgres psql -c "DROP DATABASE IF EXISTS \"$db\";" 2>/dev/null || true
58 - done
59 - }
60 -
61 - # Pre-cleanup
62 - cleanup_test_dbs
63 -
64 - # Step 1: Compilation check
65 - run_step "cargo check" cargo check --features fast-tests
66 -
67 - # Step 2: Unit tests
68 - if [ -n "$FILTER" ]; then
69 - run_step "cargo test --lib ($FILTER)" cargo test --features fast-tests --lib "$FILTER"
70 - else
71 - run_step "cargo test --lib" cargo test --features fast-tests --lib
72 - fi
73 -
74 - # Step 3: Integration tests
75 - if [ -n "$FILTER" ]; then
76 - run_step "cargo test --test integration ($FILTER)" cargo test --features fast-tests --test integration "$FILTER" -- --test-threads=8
77 - else
78 - run_step "cargo test --test integration" cargo test --features fast-tests --test integration -- --test-threads=8
79 - fi
80 -
81 - # Step 4: Clippy
82 - run_step "cargo clippy" cargo clippy --features fast-tests --all-targets -- -D warnings
83 -
84 - # Step 5: Security audit (optional)
85 - if command -v cargo-audit &>/dev/null; then
86 - # Ignored advisories + rationale live in .cargo/audit.toml (each is a
87 - # transitive advisory we can't fix by bumping our own deps). Directly-fixable
88 - # advisories are fixed in Cargo.toml, never parked in the ignore list.
89 - run_step "cargo audit" cargo audit
90 - else
91 - echo ""
92 - echo "[skip] cargo-audit not installed (cargo install cargo-audit)"
93 - fi
94 -
95 - # Step 5b: Supply-chain gate (optional) — advisories + duplicate-version +
96 - # source policy via cargo-deny (config in deny.toml). Complements cargo audit:
97 - # `bans` surfaces duplicate-version clusters, `sources` fails on any dep from an
98 - # unsanctioned registry/git remote. Licenses are excluded from the gate until
99 - # the first-party path crates declare a license field (see deny.toml).
100 - if command -v cargo-deny &>/dev/null; then
101 - run_step "cargo deny" cargo deny check advisories bans sources
102 - else
103 - echo ""
104 - echo "[skip] cargo-deny not installed (cargo install cargo-deny --locked)"
105 - fi
106 -
107 - # Post-cleanup
108 - cleanup_test_dbs
109 -
110 - # Summary
111 - echo ""
112 - echo "========================================"
113 - echo " CI Summary"
114 - echo "========================================"
115 - echo ""
116 -
117 - for step in "${PASS[@]}"; do
118 - echo " PASS $step"
119 - done
120 - for step in "${FAILURES[@]+"${FAILURES[@]}"}"; do
121 - echo " FAIL $step"
122 - done
123 -
124 - echo ""
125 - if [ ${#FAILURES[@]} -eq 0 ]; then
126 - echo "All steps passed."
127 - exit 0
128 - else
129 - echo "${#FAILURES[@]} step(s) failed."
130 - exit 1
131 - fi
D server/deploy/setup-ci.sh -66
@@ -1,66 +0,0 @@
1 - #!/bin/bash
2 - # One-time CI setup: generate SSH key for git user on Hetzner,
3 - # install public key on astra, deploy hook and CI script.
4 - #
5 - # Run from MNW/server/ on the MacBook.
6 -
7 - set -e
8 -
9 - HETZNER="root@100.120.174.96"
10 - HETZNER_SSH="-p 2200"
11 - ASTRA="max@100.106.221.39"
12 - HOOK_SRC="deploy/post-receive-hook.sh"
13 - CI_SRC="deploy/ci-on-push.sh"
14 -
15 - echo "=== CI Setup ==="
16 -
17 - # Step 1: Generate SSH key for git user on Hetzner (if not exists)
18 - echo "[1/5] Generating SSH key for git user on Hetzner..."
19 - ssh $HETZNER_SSH $HETZNER bash -s <<'REMOTE'
20 - if [ ! -f /opt/git/.ssh/id_ed25519 ]; then
21 - sudo -u git ssh-keygen -t ed25519 -f /opt/git/.ssh/id_ed25519 -N "" -C "git@hetzner-ci"
22 - chown git:git /opt/git/.ssh/id_ed25519 /opt/git/.ssh/id_ed25519.pub
23 - echo "Key generated."
24 - else
25 - echo "Key already exists."
26 - fi
27 - cat /opt/git/.ssh/id_ed25519.pub
28 - REMOTE
29 -
30 - # Step 2: Get the public key
31 - echo ""
32 - echo "[2/5] Retrieving public key..."
33 - PUBKEY=$(ssh $HETZNER_SSH $HETZNER "cat /opt/git/.ssh/id_ed25519.pub")
34 - echo " $PUBKEY"
35 -
36 - # Step 3: Install public key on astra
37 - echo ""
38 - echo "[3/5] Installing public key on astra..."
39 - echo " Copy this key to astra's ~/.ssh/authorized_keys:"
40 - echo ""
41 - echo " $PUBKEY"
42 - echo ""
43 - echo " Run on astra: echo '$PUBKEY' >> /home/max/.ssh/authorized_keys"
44 - echo ""
45 - read -p " Press Enter once the key is installed on astra..."
46 -
47 - # Step 4: Deploy post-receive hook
48 - echo ""
49 - echo "[4/5] Deploying post-receive hook..."
50 - scp $HETZNER_SSH $HOOK_SRC $HETZNER:/opt/git/max/makenotwork.git/hooks/post-receive
51 - ssh $HETZNER_SSH $HETZNER "chown git:git /opt/git/max/makenotwork.git/hooks/post-receive && chmod +x /opt/git/max/makenotwork.git/hooks/post-receive"
52 - echo " Hook installed."
53 -
54 - # Step 5: Deploy CI script to astra
55 - echo ""
56 - echo "[5/5] Deploying CI script to astra..."
57 - echo " Copy $CI_SRC to astra:/home/max/staging/ci-on-push.sh"
58 - echo " (Cannot scp to astra from here if SSH is down — copy manually)"
59 - echo ""
60 - echo " scp $CI_SRC $ASTRA:/home/max/staging/ci-on-push.sh"
61 - echo " ssh $ASTRA 'chmod +x /home/max/staging/ci-on-push.sh'"
62 -
63 - echo ""
64 - echo "=== Setup Complete ==="
65 - echo ""
66 - echo "Test: push a commit to mnw/main and check WAM for a CI ticket."
D server/docs/cicd.md -201
@@ -1,201 +0,0 @@
1 - # CI/CD Pipeline
2 -
3 - ## Overview
4 -
5 - Push-triggered CI on astra, WAM ticket notifications, staged promotion to Hetzner production. Monitored via a ratatui TUI on the tailnet.
6 -
7 - ## Architecture
8 -
9 - ```
10 - MacBook (dev)
11 - |
12 - | git push mnw main
13 - v
14 - Hetzner (production, x86_64)
15 - |
16 - | post-receive hook: ssh astra
17 - v
18 - Astra (CI/build, aarch64 Linux)
19 - |
20 - | 1. cargo check
21 - | 2. cargo test --lib
22 - | 3. cargo test --test integration
23 - | 4. cargo clippy
24 - | 5. cargo audit
25 - |
26 - | on success:
27 - | - WAM ticket (pass, test count, duration)
28 - | - cross-compile x86_64 binary (cargo-zigbuild)
29 - | - scp binary to Hetzner staging dir
30 - |
31 - | on failure:
32 - | - WAM ticket (fail, which step, error)
33 - | - stop (do not build or promote)
34 - |
35 - v
36 - Hetzner staging (/opt/makenotwork-staging/)
37 - |
38 - | PoM health checks staging instance on port 3001
39 - |
40 - | manual or TUI-driven promote
41 - v
42 - Hetzner production (/opt/makenotwork/)
43 - ```
44 -
45 - ## Machines
46 -
47 - | Machine | Role | Arch | Tailscale IP |
48 - |---------|------|------|-------------|
49 - | MacBook | Development | aarch64 macOS | local |
50 - | Astra | CI runner, native aarch64 + cross-compile x86_64 | aarch64 Linux | 100.106.221.39 |
51 - | Hetzner | Production server, staging host | x86_64 Linux | 100.120.174.96 |
52 -
53 - Astra is aarch64 Linux. It runs tests natively and cross-compiles for x86_64 (Hetzner) via cargo-zigbuild.
54 -
55 - ---
56 -
57 - ## Step 1: Push-triggered CI -- IMPLEMENTED (pending astra SSH)
58 -
59 - **Scripts:**
60 - - `deploy/post-receive-hook.sh` -- git hook for Hetzner (triggers CI on main push, OTA on tag push)
61 - - `deploy/ci-on-push.sh` -- wrapper on astra (pulls, runs CI, reports to WAM)
62 - - `deploy/setup-ci.sh` -- one-time provisioning (SSH key, hook install)
63 -
64 - **Trigger:** Git post-receive hook on Hetzner SSHes to astra on `refs/heads/main` push.
65 -
66 - **What runs:** `/home/max/staging/ci-on-push.sh` calls existing `run-ci.sh`:
67 - - cargo check
68 - - cargo test --lib (unit tests)
69 - - cargo test --test integration (8 threads, isolated per-test databases)
70 - - cargo clippy
71 - - cargo audit
72 -
73 - **Notification:** WAM ticket created with pass/fail, step counts, duration, last 30 lines of output.
74 -
75 - **Setup required:**
76 - 1. Generate SSH key for git user on Hetzner (`deploy/setup-ci.sh` handles this)
77 - 2. Install public key on astra `~/.ssh/authorized_keys`
78 - 3. Copy `ci-on-push.sh` to astra `/home/max/staging/`
79 - 4. Install hook on Hetzner (setup script handles this)
80 - 5. Astra needs: `WAM_URL=http://100.120.174.96:7890` in environment
81 -
82 - ---
83 -
84 - ## Step 2: Auto-build on CI success
85 -
86 - On CI pass, `ci-on-push.sh` continues:
87 - 1. Cross-compile x86_64 release binary on astra (`cargo zigbuild --release --target x86_64-unknown-linux-gnu`)
88 - 2. scp binary to Hetzner `/opt/makenotwork-staging/makenotwork`
89 - 3. WAM ticket: "build ready for staging"
90 -
91 - Astra prerequisites:
92 - - `rustup target add x86_64-unknown-linux-gnu`
93 - - `cargo install cargo-zigbuild` (or system zig + zigbuild)
94 -
95 - ---
96 -
97 - ## Step 3: Staging environment
98 -
99 - Hetzner runs a second instance for validation before production:
100 - - Binary at `/opt/makenotwork-staging/makenotwork`
101 - - Systemd unit: `makenotwork-staging.service` (port 3001)
102 - - Database: `makenotwork_staging` (separate from production)
103 - - Env: copy of production `.env` with different port and DB URL
104 - - PoM health-checks staging at `http://100.120.174.96:3001/api/health`
105 - - WAM ticket on staging health pass: "staging healthy, ready to promote"
106 -
107 - ---
108 -
109 - ## Step 4: Promote to production
110 -
111 - `deploy/promote.sh`:
112 - 1. Send 30s restart warning via internal API
113 - 2. Stop production service
114 - 3. Copy staging binary to `/opt/makenotwork/makenotwork`
115 - 4. Start production service
116 - 5. PoM verifies health within 30s
117 - 6. If unhealthy: rollback (restore previous binary from `/opt/makenotwork/makenotwork.prev`, restart)
118 - 7. WAM ticket: promoted or rolled back
119 -
120 - ---
121 -
122 - ## Step 5: CI/CD Monitor TUI
123 -
124 - A ratatui-based TUI (part of WAM or standalone) for real-time pipeline visibility across the tailnet. Connects to WAM's HTTP API to display CI/CD state.
125 -
126 - ### Views
127 -
128 - **Pipeline dashboard (default):**
129 - ```
130 - CI/CD v0.4.0 -> v0.4.1
131 - ---------------------------------------------------------------
132 - [v] check [v] test-lib [v] test-int [v] clippy [v] audit
133 - [v] build [v] staging [ ] promote
134 - ---------------------------------------------------------------
135 - Last CI: 2m ago (42s, 715 tests passed)
136 - Staging: healthy (3m uptime)
137 - Prod: v0.4.0 (2h uptime)
138 - ```
139 -
140 - **Recent runs:**
141 - - List of recent CI runs with pass/fail, duration, commit hash
142 - - Filter by branch, status
143 - - Enter to view full log
144 -
145 - **Actions (keyboard-driven):**
146 - - `p` -- promote staging to production (with confirmation)
147 - - `r` -- re-run CI manually (SSH to astra)
148 - - `s` -- view staging health details
149 - - `l` -- view full CI log
150 - - `d` -- deploy current staging binary (skip CI)
151 -
152 - **Data sources:**
153 - - WAM tickets (source: `ci`) for CI run results
154 - - WAM tickets (source: `build-failed`, `health-status-change`) for problems
155 - - PoM API for staging + production health status
156 - - SSH to astra for live CI log streaming (if a run is in progress)
157 -
158 - ### Implementation
159 -
160 - Could be:
161 - 1. A new `wam pipeline` subcommand (simplest, reuses WAM's DB and TUI infrastructure)
162 - 2. A new `wam/src/pipeline.rs` view that queries WAM tickets filtered by `source: ci`
163 - 3. A standalone binary if the scope grows beyond CI/CD
164 -
165 - ---
166 -
167 - ## Existing Infrastructure
168 -
169 - | Component | Location | Status |
170 - |-----------|----------|--------|
171 - | CI runner | astra:/home/max/staging/run-ci.sh | exists |
172 - | CI trigger | deploy/ci-on-push.sh | written, needs deploy to astra |
173 - | Post-receive hook | deploy/post-receive-hook.sh | written, needs install |
174 - | Setup script | deploy/setup-ci.sh | written, needs astra SSH |
175 - | Test database | astra PostgreSQL | exists |
176 - | Deploy script | deploy/deploy.sh | exists, manual |
177 - | PoM monitoring | astra + hetzner (peer mesh) | exists, 5min interval |
178 - | WAM tickets | hetzner:7890 | running |
179 - | WAM TUI | wam binary | running, needs pipeline view |
180 -
181 - ## Astra CI Environment
182 -
183 - Required:
184 - - Rust toolchain (stable) with `x86_64-unknown-linux-gnu` target
185 - - cargo-zigbuild + zig (for x86_64 cross-compilation)
186 - - PostgreSQL running (test databases created/dropped per test)
187 - - SSH access to Hetzner port 2200 (for scp binary upload)
188 - - `WAM_URL=http://100.120.174.96:7890` in environment or `.bashrc`
189 - - `SQLX_OFFLINE=true` in environment
190 - - `TEST_DATABASE_URL=postgres:///postgres` in environment
191 -
192 - ## File Inventory
193 -
194 - ```
195 - server/deploy/
196 - deploy.sh # existing manual deploy (cross-compile + upload + restart)
197 - run-ci.sh # existing CI script (check, test, clippy, audit)
198 - ci-on-push.sh # NEW: pull + CI + WAM ticket wrapper
199 - post-receive-hook.sh # NEW: git hook (CI trigger on main, OTA on tags)
200 - setup-ci.sh # NEW: one-time provisioning (SSH key, hook install)
201 - ```