Skip to main content

max / makenotwork

chore: move server/deploy docs to private store Contains infra-sensitive ops docs (SSH hosts, Hetzner setup, rollback/recovery procedures, pre-launch testing checklist).
Author: Max J. <87768334+MaxJMath@users.noreply.github.com> · 2026-05-21 01:55 UTC
Commit: 3883890f82e169aec183f65fc208a5f4f549ee7e
Parent: 08cbfab
6 files changed, +0 insertions, -1417 deletions
D server/deploy/README.md -156
@@ -1,156 +0,0 @@
1 - # MNW Deployment
2 -
3 - Scripts and configuration for deploying MNW to production.
4 -
5 - ## Quick Reference
6 -
7 - ```sh
8 - ./deploy/deploy.sh # Full: build + config + binary + restart
9 - ./deploy/deploy.sh --quick # Build + binary + restart (skip config)
10 - ./deploy/deploy.sh --config # Config files only (Caddyfile, systemd, static, docs)
11 - ```
12 -
13 - Run all commands from the `MNW/` directory.
14 -
15 - ## Prerequisites (one-time)
16 -
17 - ```sh
18 - brew install zig
19 - cargo install cargo-zigbuild
20 - rustup target add x86_64-unknown-linux-gnu
21 - ```
22 -
23 - ## What Each Mode Does
24 -
25 - ### Full deploy (default)
26 -
27 - 1. Cross-compiles the binary with `cargo zigbuild --release --target x86_64-unknown-linux-gnu`
28 - 2. Uploads config files (Caddyfile, systemd unit, error pages, security configs)
29 - 3. Minifies CSS via `clean-css-cli`, uploads static assets via rsync
30 - 4. Uploads public site-docs and generated rustdoc
31 - 5. Sends a 30-second restart warning to connected users via internal API
32 - 6. Stops the service, uploads the binary (+ `mnw-admin` if present), restarts
33 - 7. Verifies the app responds on `http://127.0.0.1:3000`
34 -
35 - ### Quick deploy (`--quick`)
36 -
37 - Skips config/static/docs upload. Builds, warns users, uploads binary, restarts.
38 -
39 - ### Config deploy (`--config`)
40 -
41 - Uploads Caddyfile, systemd unit, error pages, security configs, minified CSS, static assets, site-docs, and rustdoc. Reloads systemd and restarts Caddy. Does not touch the application binary.
42 -
43 - ## Production Server
44 -
45 - - **Host:** Hetzner VPS (CCX13 x86, US-West)
46 - - **Public IP:** `5.78.144.244`
47 - - **Tailscale IP:** `100.120.174.96` (hostname: `alpha-west-1`)
48 - - **SSH:** `root@100.120.174.96` (via Tailscale only)
49 - - **OS:** Ubuntu, x86_64
50 -
51 - ### Filesystem layout
52 -
53 - ```
54 - /opt/makenotwork/
55 - makenotwork Application binary
56 - mnw-admin Admin CLI binary
57 - .env Environment variables (secrets)
58 - static/ CSS, JS, fonts, images
59 - error-pages/ Custom 404/500/502 pages
60 - backup-db.sh Database backup script
61 - docs/public/ Site documentation (rendered by DocEngine)
62 - rustdoc/ Generated API reference
63 - deploy/ Security config copies
64 -
65 - /opt/git/ Bare git repos (source browser)
66 - makenotwork.git/
67 - synckit-client.git/
68 - ...
69 -
70 - /etc/caddy/Caddyfile Reverse proxy config
71 - /etc/caddy/cloudflare-origin.pem Cloudflare Origin CA cert
72 - /etc/caddy/cloudflare-origin-key.pem Origin CA private key
73 - /etc/caddy/cloudflare-authenticated-origin-pull-ca.pem Cloudflare AOP CA
74 - /etc/caddy/maxj-phd-origin.pem maxj.phd Origin CA cert
75 - /etc/caddy/maxj-phd-origin-key.pem maxj.phd Origin CA key
76 - /etc/systemd/system/makenotwork.service systemd unit
77 - ```
78 -
79 - ### Services
80 -
81 - | Service | Role | Port |
82 - |---------|------|------|
83 - | `makenotwork` | Application (systemd, runs as `makenotwork` user) | 3000 |
84 - | `caddy` | Reverse proxy, TLS termination | 443 |
85 - | `postgresql` | Database (`makenotwork` db + user) | 5432 |
86 -
87 - ### Networking
88 -
89 - - **Cloudflare** proxies all HTTP/HTTPS traffic (origin IP hidden)
90 - - **SSL:** Full (Strict) mode with Cloudflare Origin CA (15yr wildcard for `*.makenot.work` and `*.maxj.phd`)
91 - - **Authenticated Origin Pulls** enabled (mTLS between Cloudflare and origin)
92 - - **SSH:** `ssh.makenot.work` DNS A record points directly to public IP (proxy OFF) for git push/pull
93 - - **Firewall:** ufw + fail2ban, sshd hardened
94 -
95 - ## Scripts Reference
96 -
97 - | Script | Purpose |
98 - |--------|---------|
99 - | `deploy.sh` | Main deployment script (build, upload, restart) |
100 - | `backup-db.sh` | PostgreSQL backup (pg_dump, uploaded to server) |
101 - | `generate-rustdoc.sh` | Generate rustdoc for library crates |
102 - | `ota-publish.sh` | Publish OTA release (auth, create release, presigned upload, verify) |
103 - | `run-ci.sh` | CI runner (check, test, clippy, audit) -- runs on astra |
104 - | `setup-firewall.sh` | Configure ufw rules |
105 - | `setup-git-ssh.sh` | Configure git SSH access |
106 - | `setup-ssh-keys.sh` | Deploy SSH authorized keys |
107 -
108 - ## Configuration Files
109 -
110 - | File | Deployed to | Purpose |
111 - |------|-------------|---------|
112 - | `Caddyfile` | `/etc/caddy/Caddyfile` | Reverse proxy rules for all domains |
113 - | `makenotwork.service` | `/etc/systemd/system/` | systemd unit (EnvironmentFile, restart policy) |
114 - | `env.production` | Reference for `.env` format | **Not deployed** -- `.env` is edited on server |
115 - | `fail2ban-sshd.conf` | `/opt/makenotwork/deploy/` | fail2ban jail config |
116 - | `sshd-git.conf` | `/opt/makenotwork/deploy/` | SSH config for git user |
117 - | `error-pages/*.html` | `/opt/makenotwork/error-pages/` | Custom Caddy error pages |
118 -
119 - ## Environment Variables
120 -
121 - All secrets live in `/opt/makenotwork/.env` (loaded by systemd `EnvironmentFile`). See `env.production` for the template. Key variables:
122 -
123 - - `DATABASE_URL` -- PostgreSQL connection string
124 - - `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_CLIENT_ID` -- Stripe Connect
125 - - `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET` -- Hetzner Object Storage
126 - - `POSTMARK_SERVER_TOKEN` -- transactional email
127 - - `JWT_SECRET`, `SESSION_SECRET` -- authentication
128 - - `SYNCKIT_JWT_SECRET` -- SyncKit token signing
129 - - `HOST_URL` -- public base URL (`https://makenot.work`)
130 -
131 - ## Astra (dev/test server)
132 -
133 - - **Tailscale IP:** `100.106.221.39`
134 - - **OS:** Pop!_OS 24.04 LTS, aarch64 (96 cores, 125GB RAM)
135 - - **PostgreSQL 16** with tuned settings
136 - - **Used for:** CI, integration tests, staging
137 -
138 - ### Running tests on astra
139 -
140 - ```sh
141 - ssh 100.106.221.39
142 - /home/max/staging/run-tests.sh # All tests
143 - /home/max/staging/run-tests.sh auth:: # Filtered
144 - ```
145 -
146 - Use `--test-threads=8` (or the `RUST_TEST_THREADS=8` env var set in `.bashrc`) to avoid overwhelming PostgreSQL with 96 concurrent `CREATE DATABASE` calls.
147 -
148 - ## Versioning
149 -
150 - - Version is set in `Cargo.toml` and compiled into the binary via `env!("CARGO_PKG_VERSION")`
151 - - **Always ask before bumping** -- never auto-increment
152 - - Edit `Cargo.toml` version field before building
153 -
154 - ## Full Setup
155 -
156 - See `SERVER_SETUP.md` for the complete provisioning checklist (PostgreSQL, Caddy, systemd, Stripe, S3, DNS, Cloudflare, security hardening) and `RECOVERY.md` for disaster recovery procedures.
D server/deploy/RECOVERY.md -224
@@ -1,224 +0,0 @@
1 - # Database Recovery Procedure
2 -
3 - How to restore the Makenotwork database from a backup.
4 -
5 - Backups are gzipped SQL dumps kept for 30 days in two locations:
6 -
7 - - **Primary (Hetzner):** `/opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz`
8 - - **Offsite (astra):** `/opt/backups/mnw/makenotwork-YYYYMMDD-HHMMSS.sql.gz` (synced after each backup via Tailscale)
9 -
10 - If Hetzner is destroyed, the offsite copy on astra survives.
11 -
12 - ---
13 -
14 - ## List Available Backups
15 -
16 - ```bash
17 - ls -lh /opt/makenotwork/backups/makenotwork-*.sql.gz
18 - ```
19 -
20 - ## Full Restore
21 -
22 - Replaces the entire database with the backup contents.
23 -
24 - ### 1. Stop the application
25 -
26 - ```bash
27 - sudo systemctl stop makenotwork
28 - ```
29 -
30 - ### 2. Drop and recreate the database
31 -
32 - ```bash
33 - sudo -u postgres psql <<EOF
34 - DROP DATABASE makenotwork;
35 - CREATE DATABASE makenotwork OWNER makenotwork;
36 - EOF
37 - ```
38 -
39 - ### 3. Restore from backup
40 -
41 - ```bash
42 - gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz \
43 - | psql -U makenotwork -d makenotwork
44 - ```
45 -
46 - ### 4. Verify
47 -
48 - ```bash
49 - psql -U makenotwork -d makenotwork -c "SELECT COUNT(*) FROM users;"
50 - psql -U makenotwork -d makenotwork -c "SELECT COUNT(*) FROM projects;"
51 - psql -U makenotwork -d makenotwork -c "SELECT COUNT(*) FROM items;"
52 - ```
53 -
54 - ### 5. Restart the application
55 -
56 - ```bash
57 - sudo systemctl start makenotwork
58 - sudo systemctl status makenotwork
59 - ```
60 -
61 - ### 6. Smoke test
62 -
63 - - Visit https://makenot.work/ and confirm it loads
64 - - Check /health for system status
65 - - Try logging in
66 -
67 - ---
68 -
69 - ## Selective Restore (Single Table)
70 -
71 - If only one table is corrupted, extract and restore it without touching the rest.
72 -
73 - ### 1. Extract the table from the backup
74 -
75 - ```bash
76 - gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz \
77 - | grep -A9999999 "^COPY public.TABLE_NAME" \
78 - | sed '/^\\\.$/q' > /tmp/table_restore.sql
79 - ```
80 -
81 - ### 2. Review the extracted data
82 -
83 - ```bash
84 - head -20 /tmp/table_restore.sql
85 - wc -l /tmp/table_restore.sql
86 - ```
87 -
88 - ### 3. Clear and restore the table
89 -
90 - ```bash
91 - psql -U makenotwork -d makenotwork -c "DELETE FROM TABLE_NAME;"
92 - psql -U makenotwork -d makenotwork < /tmp/table_restore.sql
93 - ```
94 -
95 - **Note:** Watch for foreign key constraints. If the table has dependencies, you may need to temporarily disable triggers:
96 -
97 - ```bash
98 - psql -U makenotwork -d makenotwork <<EOF
99 - SET session_replication_role = 'replica';
100 - DELETE FROM TABLE_NAME;
101 - \i /tmp/table_restore.sql
102 - SET session_replication_role = 'origin';
103 - EOF
104 - ```
105 -
106 - ---
107 -
108 - ## Restore to a Separate Database (For Inspection)
109 -
110 - Useful when you want to check backup contents without touching production.
111 -
112 - ```bash
113 - sudo -u postgres createdb makenotwork_restore -O makenotwork
114 - gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz \
115 - | psql -U makenotwork -d makenotwork_restore
116 -
117 - # Inspect
118 - psql -U makenotwork -d makenotwork_restore
119 -
120 - # Clean up when done
121 - sudo -u postgres dropdb makenotwork_restore
122 - ```
123 -
124 - ---
125 -
126 - ## Failure Scenarios
127 -
128 - ### Application won't start after restore
129 -
130 - Check migration state. The backup includes the `_sqlx_migrations` table, so the app should recognize the schema. If migrations are ahead of the backup:
131 -
132 - ```bash
133 - # Check what the app expects vs what's in the DB
134 - psql -U makenotwork -d makenotwork \
135 - -c "SELECT version, description FROM _sqlx_migrations ORDER BY version;"
136 - ```
137 -
138 - If the backup is from before a migration was applied, the app will attempt to run pending migrations on startup.
139 -
140 - ### Backup file is corrupted
141 -
142 - ```bash
143 - # Test gzip integrity
144 - gzip -t /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz
145 - ```
146 -
147 - If the most recent backup is bad, use the previous day's backup.
148 -
149 - ### Hetzner destroyed — restore from offsite
150 -
151 - If the Hetzner VPS is lost, backups survive on astra:
152 -
153 - ```bash
154 - # From astra, list available backups
155 - ls -lh /opt/backups/mnw/makenotwork-*.sql.gz
156 -
157 - # Copy the latest to the new server
158 - scp /opt/backups/mnw/makenotwork-YYYYMMDD-HHMMSS.sql.gz \
159 - root@<new-server>:/opt/makenotwork/backups/
160 - ```
161 -
162 - Then follow the Full Restore procedure above on the new server.
163 -
164 - ### No backups available
165 -
166 - If all backups have been lost, the only option is to start fresh:
167 -
168 - ```bash
169 - sudo -u postgres psql <<EOF
170 - DROP DATABASE makenotwork;
171 - CREATE DATABASE makenotwork OWNER makenotwork;
172 - EOF
173 - sudo systemctl restart makenotwork
174 - # The app will run all migrations and create a clean schema
175 - ```
176 -
177 - ---
178 -
179 - ## Backup Verification
180 -
181 - To confirm backups are running and healthy:
182 -
183 - ```bash
184 - # Check the most recent backup
185 - ls -lt /opt/makenotwork/backups/makenotwork-*.sql.gz | head -1
186 -
187 - # Check backup log for errors
188 - tail -20 /opt/makenotwork/backups/backup.log
189 -
190 - # Check cron is scheduled
191 - sudo crontab -u makenotwork -l
192 - ```
193 -
194 - ---
195 -
196 - ## Monthly Restore Test
197 -
198 - Run once per month to verify backups are actually restorable. Use the offsite copy on astra to avoid touching production.
199 -
200 - ```bash
201 - # On astra — restore latest backup to a scratch database
202 - LATEST=$(ls -t /opt/backups/mnw/makenotwork-*.sql.gz | head -1)
203 - sudo -u postgres createdb mnw_restore_test -O postgres
204 - gunzip -c "$LATEST" | psql -U postgres -d mnw_restore_test -q
205 -
206 - # Verify row counts (should be non-zero)
207 - psql -U postgres -d mnw_restore_test -c "SELECT 'users' AS t, COUNT(*) FROM users UNION ALL SELECT 'projects', COUNT(*) FROM projects UNION ALL SELECT 'items', COUNT(*) FROM items UNION ALL SELECT 'transactions', COUNT(*) FROM transactions;"
208 -
209 - # Verify migration state
210 - psql -U postgres -d mnw_restore_test -c "SELECT COUNT(*) AS migrations FROM _sqlx_migrations;"
211 -
212 - # Clean up
213 - sudo -u postgres dropdb mnw_restore_test
214 -
215 - # Log result
216 - echo "$(date -u +%Y-%m-%d) restore-test OK: $LATEST" >> /opt/backups/mnw/restore-test.log
217 - ```
218 -
219 - If any step fails, investigate immediately — a backup that can't be restored isn't a backup.
220 -
221 - Schedule via cron on astra (first of each month):
222 - ```
223 - 0 4 1 * * /opt/backups/mnw/test-restore.sh >> /opt/backups/mnw/restore-test.log 2>&1
224 - ```
D server/deploy/ROLLBACK.md -207
@@ -1,207 +0,0 @@
1 - # Rollback Guide — MNW Server
2 -
3 - ## Quick Rollback (Re-deploy Previous Binary)
4 -
5 - The previous binary is overwritten during deploy, so rollback means re-building a previous commit and deploying it.
6 -
7 - ### Steps
8 -
9 - 1. **Identify the last known-good commit:**
10 - ```bash
11 - cd MNW/server
12 - git log --oneline -10
13 - ```
14 -
15 - 2. **Check out and build the previous version:**
16 - ```bash
17 - git checkout <commit-hash>
18 - cargo zigbuild --release --target x86_64-unknown-linux-gnu
19 - ```
20 -
21 - 3. **Deploy the rollback binary:**
22 - ```bash
23 - ssh root@100.120.174.96 "systemctl stop makenotwork || true"
24 - scp target/x86_64-unknown-linux-gnu/release/makenotwork root@100.120.174.96:/opt/makenotwork/makenotwork
25 - ssh root@100.120.174.96 "chmod +x /opt/makenotwork/makenotwork"
26 - ssh root@100.120.174.96 "systemctl start makenotwork"
27 - ```
28 -
29 - 4. **Verify:**
30 - ```bash
31 - ssh root@100.120.174.96 "systemctl status makenotwork --no-pager"
32 - ssh root@100.120.174.96 "curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:3000"
33 - ```
34 -
35 - 5. **Return to main branch:**
36 - ```bash
37 - git checkout main
38 - ```
39 -
40 - ## Emergency Stop
41 -
42 - Stop the application immediately. Caddy will serve the 502 error page (auto-retries every 10 seconds).
43 -
44 - ```bash
45 - ssh root@100.120.174.96 "systemctl stop makenotwork"
46 - ```
47 -
48 - To restart:
49 - ```bash
50 - ssh root@100.120.174.96 "systemctl start makenotwork"
51 - ```
52 -
53 - ## Database Restore
54 -
55 - Backups are created daily at 03:00 UTC by cron (`/opt/makenotwork/backup-db.sh`). Stored at `/opt/makenotwork/backups/`, 30-day retention.
56 -
57 - ### Restore from backup
58 -
59 - 1. **Stop the application:**
60 - ```bash
61 - ssh root@100.120.174.96 "systemctl stop makenotwork"
62 - ```
63 -
64 - 2. **List available backups:**
65 - ```bash
66 - ssh root@100.120.174.96 "ls -lh /opt/makenotwork/backups/"
67 - ```
68 -
69 - 3. **Create a backup of the current (broken) state first:**
70 - ```bash
71 - ssh root@100.120.174.96 "sudo -u makenotwork pg_dump makenotwork | gzip > /opt/makenotwork/backups/makenotwork-pre-restore.sql.gz"
72 - ```
73 -
74 - 4. **Drop and recreate the database:**
75 - ```bash
76 - ssh root@100.120.174.96 "sudo -u postgres psql -c 'DROP DATABASE makenotwork;'"
77 - ssh root@100.120.174.96 "sudo -u postgres psql -c \"CREATE DATABASE makenotwork OWNER makenotwork;\""
78 - ```
79 -
80 - 5. **Restore from backup:**
81 - ```bash
82 - ssh root@100.120.174.96 "gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz | sudo -u makenotwork psql makenotwork"
83 - ```
84 -
85 - 6. **Restart the application** (migrations will run on boot and apply any missing ones):
86 - ```bash
87 - ssh root@100.120.174.96 "systemctl start makenotwork"
88 - ```
89 -
90 - 7. **Verify:**
91 - ```bash
92 - ssh root@100.120.174.96 "systemctl status makenotwork --no-pager"
93 - ssh root@100.120.174.96 "curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:3000"
94 - ```
95 -
96 - ### Notes on DB Restore
97 -
98 - - Migrations are auto-applied on boot via sqlx. If you restore an older backup and the current binary has newer migrations, they'll run automatically.
99 - - If a migration is incompatible with the restored data, you'll need to also rollback the binary (see Quick Rollback above).
100 - - For finer-grained recovery (less than 24h of data loss), use Point-in-Time Recovery below.
101 -
102 - ## Point-in-Time Recovery (PITR)
103 -
104 - WAL continuous archiving caps data loss at ~5 minutes. Use PITR when you need to recover to a specific moment rather than the last daily dump.
105 -
106 - **Prerequisites:** WAL archiving must be enabled (see `deploy/setup-wal-archiving.sh`). WAL segments are archived to `/opt/makenotwork/wal-archive/`.
107 -
108 - ### Steps
109 -
110 - 1. **Stop the application:**
111 - ```bash
112 - ssh root@100.120.174.96 "systemctl stop makenotwork"
113 - ```
114 -
115 - 2. **Stop PostgreSQL:**
116 - ```bash
117 - ssh root@100.120.174.96 "systemctl stop postgresql"
118 - ```
119 -
120 - 3. **Back up the current (broken) data directory:**
121 - ```bash
122 - ssh root@100.120.174.96 "cp -r /var/lib/postgresql/16/main /var/lib/postgresql/16/main.broken"
123 - ```
124 -
125 - 4. **Restore the base backup** (latest daily pg_dump):
126 - ```bash
127 - ssh root@100.120.174.96 "sudo -u postgres psql -c 'DROP DATABASE makenotwork;'"
128 - ssh root@100.120.174.96 "sudo -u postgres psql -c \"CREATE DATABASE makenotwork OWNER makenotwork;\""
129 - ssh root@100.120.174.96 "gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz | sudo -u makenotwork psql makenotwork"
130 - ```
131 -
132 - 5. **Configure recovery target** (create a recovery signal file):
133 - ```bash
134 - ssh root@100.120.174.96 "cat > /var/lib/postgresql/16/main/recovery.signal <<EOF
135 - EOF"
136 - ```
137 -
138 - Add to `postgresql.conf` (or a conf.d override):
139 - ```
140 - restore_command = 'cp /opt/makenotwork/wal-archive/%f %p'
141 - recovery_target_time = '2026-05-08 14:30:00 UTC'
142 - recovery_target_action = 'promote'
143 - ```
144 -
145 - 6. **Start PostgreSQL** (it will replay WAL segments up to the target time):
146 - ```bash
147 - ssh root@100.120.174.96 "systemctl start postgresql"
148 - ```
149 -
150 - 7. **Verify recovery** (check logs for "recovery complete"):
151 - ```bash
152 - ssh root@100.120.174.96 "journalctl -u postgresql --since '5 minutes ago' --no-pager"
153 - ```
154 -
155 - 8. **Remove recovery config** (clean up the override so future restarts don't re-enter recovery):
156 - ```bash
157 - ssh root@100.120.174.96 "rm -f /var/lib/postgresql/16/main/recovery.signal"
158 - ```
159 - Remove the `restore_command`, `recovery_target_time`, and `recovery_target_action` lines from the config.
160 -
161 - 9. **Restart the application:**
162 - ```bash
163 - ssh root@100.120.174.96 "systemctl start makenotwork"
164 - ```
165 -
166 - ### When to Use PITR vs Full Dump Restore
167 -
168 - | Scenario | Use |
169 - |----------|-----|
170 - | Catastrophic corruption, need any recovery | Full dump restore (simpler) |
171 - | Need to recover to a specific point in time | PITR |
172 - | Data was accidentally deleted minutes ago | PITR |
173 - | Restoring to a known-good state from days ago | Full dump restore |
174 -
175 - ### WAL Archive Maintenance
176 -
177 - - WAL segments are pruned automatically: local (7 days via cron), offsite on astra (7 days via `sync-wal-offsite.sh`).
178 - - Daily pg_dump backups remain the primary recovery method. WAL archiving adds continuous protection on top.
179 - - Monitor WAL archive health via PoM (checks recency every hour).
180 -
181 - ## Migration Rollback
182 -
183 - Migrations are additive (new tables, new columns with defaults). There's no built-in `down` migration. If a migration causes issues:
184 -
185 - 1. Manually write a reversal SQL script
186 - 2. Apply it: `ssh root@100.120.174.96 "sudo -u makenotwork psql makenotwork < /tmp/rollback.sql"`
187 - 3. Delete the migration row from `_sqlx_migrations` so it doesn't conflict:
188 - ```sql
189 - DELETE FROM _sqlx_migrations WHERE version = <migration_number>;
190 - ```
191 -
192 - ## Service Architecture Reference
193 -
194 - - **Binary**: `/opt/makenotwork/makenotwork`
195 - - **Config**: `/opt/makenotwork/.env`
196 - - **Static**: `/opt/makenotwork/static/`
197 - - **Docs**: `/opt/makenotwork/docs/`
198 - - **Backups**: `/opt/makenotwork/backups/` (daily pg_dump)
199 - - **WAL archive**: `/opt/makenotwork/wal-archive/` (continuous)
200 - - **Systemd unit**: `/etc/systemd/system/makenotwork.service`
201 - - **Caddy config**: `/etc/caddy/Caddyfile`
202 - - **Error pages**: `/opt/makenotwork/error-pages/`
203 - - **Git repos**: `/opt/git/`
204 - - **Logs**: `journalctl -u makenotwork -f`
205 - - **Port**: 127.0.0.1:3000 (Caddy reverse proxies from 443)
206 - - **DB**: PostgreSQL `makenotwork` database, `makenotwork` user (peer auth)
207 - - **Restart policy**: `Restart=always`, `RestartSec=5`
D server/deploy/SERVER_SETUP.md -352
@@ -1,352 +0,0 @@
1 - # Makenotwork Server Setup Guide
2 -
3 - Complete checklist for deploying to Hetzner VPS.
4 -
5 - ---
6 -
7 - ## Pre-Deployment Checklist (Do Now)
8 -
9 - These can be done before provisioning the server:
10 -
11 - ### Stripe Setup
12 - - [ ] Create Stripe account (if not already)
13 - - [ ] Switch to live mode (or stay in test mode for initial testing)
14 - - [ ] Note your **Secret Key** (`sk_live_...` or `sk_test_...`)
15 - - [ ] Go to Settings > Connect settings
16 - - [ ] Note your **Client ID** (`ca_...`)
17 - - [ ] Go to Developers > Webhooks > Add endpoint
18 - - URL: `https://makenot.work/stripe/webhook`
19 - - Events: `checkout.session.completed`, `account.updated`
20 - - Note the **Webhook Secret** (`whsec_...`)
21 -
22 - ### Hetzner Object Storage Setup
23 - - [ ] Create Object Storage bucket in Hetzner Cloud Console
24 - - [ ] Bucket name: `makenotwork-files` (or your choice)
25 - - [ ] Region: `fsn1` (Frankfurt) or your preferred
26 - - [ ] Generate S3 credentials
27 - - [ ] Note: Endpoint, Access Key, Secret Key
28 -
29 - ### DNS Setup
30 - - [ ] Point `makenot.work` A record to server IP
31 - - [ ] Point `www.makenot.work` A record to server IP (for redirect)
32 -
33 - ### Generate Secrets
34 - Run locally and save for later:
35 - ```bash
36 - # JWT Secret
37 - openssl rand -base64 32
38 -
39 - # Session Secret
40 - openssl rand -base64 32
41 -
42 - # Database Password
43 - openssl rand -base64 24
44 - ```
45 -
46 - ---
47 -
48 - ## Server Provisioning
49 -
50 - ### 1. Create Hetzner VPS — DONE
51 - - Type: CCX13 x86 (US-West)
52 - - Disk: 80GB + 10GB
53 - - IP: `5.78.144.244`
54 - - DNS: Cloudflare pointing makenot.work + maxj.phd to this IP
55 -
56 - ### 2. Initial Server Setup
57 - ```bash
58 - # SSH into server
59 - ssh root@100.120.174.96
60 -
61 - # Update system
62 - apt update && apt upgrade -y
63 -
64 - # Set timezone
65 - timedatectl set-timezone America/New_York # or your timezone
66 -
67 - # Create non-root user (optional but recommended)
68 - adduser makenotwork
69 - usermod -aG sudo makenotwork
70 - ```
71 -
72 - ### 3. Install PostgreSQL
73 - ```bash
74 - # Install PostgreSQL
75 - apt install postgresql postgresql-contrib -y
76 -
77 - # Start and enable
78 - systemctl start postgresql
79 - systemctl enable postgresql
80 -
81 - # Create database and user
82 - sudo -u postgres psql << EOF
83 - CREATE USER makenotwork WITH PASSWORD '<DB_PASSWORD>';
84 - CREATE DATABASE makenotwork OWNER makenotwork;
85 - GRANT ALL PRIVILEGES ON DATABASE makenotwork TO makenotwork;
86 - EOF
87 -
88 - # Test connection
89 - psql -U makenotwork -h localhost -d makenotwork
90 - ```
91 -
92 - ### 4. Install Caddy
93 - ```bash
94 - # Install Caddy
95 - apt install -y debian-keyring debian-archive-keyring apt-transport-https
96 - curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
97 - curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
98 - apt update
99 - apt install caddy -y
100 -
101 - # Create log directory
102 - mkdir -p /var/log/caddy
103 - chown caddy:caddy /var/log/caddy
104 - ```
105 -
106 - ### 5. Create Application Directory
107 - ```bash
108 - # Create directories
109 - mkdir -p /opt/makenotwork/docs
110 - chown -R makenotwork:makenotwork /opt/makenotwork
111 -
112 - # If using root for deployment initially:
113 - # mkdir -p /opt/makenotwork/docs
114 - ```
115 -
116 - ### 6. Upload Configuration Files
117 -
118 - From your local machine:
119 - ```bash
120 - # Copy Caddyfile
121 - scp deploy/Caddyfile root@100.120.174.96:/etc/caddy/Caddyfile
122 -
123 - # Copy systemd service
124 - scp deploy/makenotwork.service root@100.120.174.96:/etc/systemd/system/
125 -
126 - # Copy environment template
127 - scp deploy/env.production root@100.120.174.96:/opt/makenotwork/.env
128 - ```
129 -
130 - ### 7. Configure Environment
131 - ```bash
132 - # SSH into server
133 - ssh root@100.120.174.96
134 -
135 - # Edit .env with your actual values
136 - nano /opt/makenotwork/.env
137 -
138 - # Secure the file
139 - chmod 600 /opt/makenotwork/.env
140 - chown makenotwork:makenotwork /opt/makenotwork/.env
141 - ```
142 -
143 - ### 8. Enable Services
144 - ```bash
145 - # Reload systemd
146 - systemctl daemon-reload
147 -
148 - # Enable services
149 - systemctl enable makenotwork
150 - systemctl enable caddy
151 -
152 - # Start Caddy (will get SSL certificate)
153 - systemctl restart caddy
154 - ```
155 -
156 - ---
157 -
158 - ## First Deployment
159 -
160 - ### Cross-Compilation Setup (one-time, on Mac)
161 - ```bash
162 - brew install zig
163 - cargo install cargo-zigbuild
164 - rustup target add x86_64-unknown-linux-gnu
165 - ```
166 -
167 - ### Build and Deploy
168 - From your local machine in the `MNW/` directory:
169 -
170 - ```bash
171 - # Make deploy script executable
172 - chmod +x deploy/deploy.sh
173 -
174 - # Deploy — cross-compiles for x86_64 Linux, uploads binary, restarts service
175 - ./deploy/deploy.sh root@100.120.174.96
176 - ```
177 -
178 - ### Verify Deployment
179 - ```bash
180 - # Check service status
181 - ssh root@100.120.174.96 "systemctl status makenotwork"
182 -
183 - # Check logs
184 - ssh root@100.120.174.96 "journalctl -u makenotwork -f"
185 -
186 - # Test endpoints
187 - curl https://makenot.work/
188 - curl https://makenot.work/docs/
189 - ```
190 -
191 - ---
192 -
193 - ## Git SSH Access
194 -
195 - Public SSH access via `git.makenot.work` for clone/push from anywhere.
196 -
197 - ### Prerequisites
198 -
199 - - `setup-git-ssh.sh` and `setup-ssh-keys.sh` already exist in `deploy/`
200 - - `mnw-admin` binary with `rebuild-keys` and `git-auth` subcommands
201 - - SSH key management UI in dashboard already functional
202 -
203 - ### 1. DNS Record
204 -
205 - Add in Cloudflare (proxy **OFF** — SSH cannot go through Cloudflare):
206 - - Type: `A`
207 - - Name: `git`
208 - - Content: `5.78.144.244`
209 - - Proxy: DNS only (grey cloud)
210 -
211 - ### 2. Create git system user
212 - ```bash
213 - ssh root@100.120.174.96
214 - bash /opt/makenotwork/deploy/setup-git-ssh.sh
215 - ```
216 -
217 - ### 3. Set up sudoers for authorized_keys rebuild
218 - ```bash
219 - bash /opt/makenotwork/deploy/setup-ssh-keys.sh
220 - ```
221 -
222 - ### 4. Install sshd config
223 - ```bash
224 - cp /opt/makenotwork/deploy/sshd-git.conf /etc/ssh/sshd_config.d/git.conf
225 - systemctl restart sshd
226 - ```
227 -
228 - ### 5. Install fail2ban
229 - ```bash
230 - apt install fail2ban -y
231 - cp /opt/makenotwork/deploy/fail2ban-sshd.conf /etc/fail2ban/jail.d/sshd.conf
232 - systemctl enable fail2ban
233 - systemctl restart fail2ban
234 - ```
235 -
236 - ### 6. Configure firewall
237 - ```bash
238 - apt install ufw -y
239 - bash /opt/makenotwork/deploy/setup-firewall.sh
240 - ```
241 -
242 - ### 7. Add GIT_SSH_HOST to .env
243 - ```bash
244 - echo 'GIT_SSH_HOST=git.makenot.work' >> /opt/makenotwork/.env
245 - systemctl restart makenotwork
246 - ```
247 -
248 - ### 8. Verify
249 - ```bash
250 - # Should print "Interactive login disabled" or similar
251 - ssh git@git.makenot.work
252 -
253 - # Clone test (after adding SSH key in dashboard)
254 - git clone git@git.makenot.work:max/makenotwork.git /tmp/test-clone
255 - rm -rf /tmp/test-clone
256 - ```
257 -
258 - ---
259 -
260 - ## Post-Deployment
261 -
262 - ### Remove Demo Data
263 - The demo seed creates a test account. Remove it:
264 - ```bash
265 - # On the server
266 - psql -U makenotwork -d makenotwork << EOF
267 - DELETE FROM transactions WHERE buyer_id IN (SELECT id FROM users WHERE email = 'elena@example.com');
268 - DELETE FROM transactions WHERE seller_id IN (SELECT id FROM users WHERE email = 'elena@example.com');
269 - DELETE FROM items WHERE project_id IN (SELECT id FROM projects WHERE user_id IN (SELECT id FROM users WHERE email = 'elena@example.com'));
270 - DELETE FROM projects WHERE user_id IN (SELECT id FROM users WHERE email = 'elena@example.com');
271 - DELETE FROM custom_links WHERE user_id IN (SELECT id FROM users WHERE email = 'elena@example.com');
272 - DELETE FROM users WHERE email = 'elena@example.com';
273 - EOF
274 - ```
275 -
276 - ### Create Your Account
277 - 1. Go to https://makenot.work/join
278 - 2. Create your account
279 - 3. Go to Dashboard > Account
280 - 4. Connect Stripe
281 - 5. Create a project and upload content
282 -
283 - ---
284 -
285 - ## Troubleshooting
286 -
287 - ### Service won't start
288 - ```bash
289 - # Check logs
290 - journalctl -u makenotwork -n 50
291 -
292 - # Common issues:
293 - # - Database connection: check DATABASE_URL
294 - # - Missing .env: check /opt/makenotwork/.env exists
295 - # - Permission denied: check file ownership
296 - ```
297 -
298 - ### SSL Certificate Issues
299 - ```bash
300 - # Check Caddy logs
301 - journalctl -u caddy -f
302 -
303 - # Verify DNS is pointing to server
304 - dig makenot.work
305 - ```
306 -
307 - ### Database Connection Failed
308 - ```bash
309 - # Test connection
310 - psql -U makenotwork -h localhost -d makenotwork
311 -
312 - # Check PostgreSQL is running
313 - systemctl status postgresql
314 -
315 - # Check pg_hba.conf allows local connections
316 - cat /etc/postgresql/*/main/pg_hba.conf | grep makenotwork
317 - ```
318 -
319 - ### Stripe Webhooks Not Working
320 - 1. Check webhook is configured in Stripe Dashboard
321 - 2. Verify URL: `https://makenot.work/stripe/webhook`
322 - 3. Check STRIPE_WEBHOOK_SECRET matches
323 - 4. Test with Stripe CLI: `stripe listen --forward-to localhost:3000/stripe/webhook`
324 -
325 - ---
326 -
327 - ## Maintenance
328 -
329 - ### Update Application
330 - ```bash
331 - ./deploy/deploy.sh root@100.120.174.96
332 - ```
333 -
334 - ### View Logs
335 - ```bash
336 - # Application logs
337 - ssh root@100.120.174.96 "journalctl -u makenotwork -f"
338 -
339 - # Caddy logs
340 - ssh root@100.120.174.96 "tail -f /var/log/caddy/makenotwork.log"
341 - ```
342 -
343 - ### Database Backups
344 - Automated daily backups with 30-day retention. See setup in `backup-db.sh` header comments.
345 -
346 - For recovery procedures, see `RECOVERY.md`.
347 -
348 - ### Restart Services
349 - ```bash
350 - ssh root@100.120.174.96 "sudo systemctl restart makenotwork"
351 - ssh root@100.120.174.96 "sudo systemctl restart caddy"
352 - ```
D server/deploy/SSH_ACCESS.md -57
@@ -1,57 +0,0 @@
1 - # SSH Access to Production Server
2 -
3 - ## Hosts
4 -
5 - | Host | Tailscale IP | Public IP | Role |
6 - |------|-------------|-----------|------|
7 - | Hetzner (prod) | `100.120.174.96` | `5.78.144.244` | Production server |
8 - | Astra (dev/test) | `100.106.221.39` | — | Build host, CI runner |
9 -
10 - ## Admin SSH (Hetzner)
11 -
12 - Regular sshd listens on **port 2200**, bound to the Tailscale interface only.
13 - Tailscale SSH is also active, allowing passwordless access for Tailscale-authenticated users.
14 -
15 - **Via Tailscale SSH (preferred):**
16 - ```bash
17 - ssh root@100.120.174.96
18 - ```
19 -
20 - **Via regular sshd (if Tailscale SSH is off):**
21 - ```bash
22 - ssh -p 2200 deploy@100.120.174.96
23 - ```
24 - Requires your public key in `~deploy/.ssh/authorized_keys` on the server.
25 -
26 - **Note:** The `max` user does not exist on the server. Use `root` (Tailscale SSH) or `deploy` (regular sshd).
27 -
28 - ## Git SSH (mnw-cli)
29 -
30 - mnw-cli runs on **port 22** on the public IP. This is for git operations and the CLI TUI, not admin access.
31 -
32 - ```bash
33 - # ~/.ssh/config entry
34 - Host mnw
35 - HostName 5.78.144.244
36 - User git
37 - ```
38 -
39 - ## Service Locations
40 -
41 - | Service | Path |
42 - |---------|------|
43 - | App binary | `/opt/makenotwork/makenotwork` |
44 - | Admin binary | `/opt/makenotwork/mnw-admin` |
45 - | Environment | `/opt/makenotwork/.env` |
46 - | Static files | `/opt/makenotwork/static/` |
47 - | Caddy config | `/etc/caddy/Caddyfile` |
48 - | Systemd unit | `/etc/systemd/system/makenotwork.service` |
49 - | DB backups | `/opt/makenotwork/backups/` |
50 - | Logs | `journalctl -u makenotwork` |
51 -
52 - ## Astra
53 -
54 - Tailscale SSH only:
55 - ```bash
56 - ssh root@100.106.221.39
57 - ```
D server/deploy/human_testing.md -421
@@ -1,421 +0,0 @@
1 - # Makenotwork — Pre-Launch Manual Testing
2 -
3 - ## How to Test
4 -
5 - - Automated tests cover units and integration (1,060+ passing) but can't catch visual bugs, broken flows, or UX issues
6 - - Work through each section sequentially, checking boxes as you go
7 - - If something fails, note the issue inline and keep going — don't block the whole run
8 - - Prioritized: P0 first (launch-blocking), then P1 (core features), then P2 (edge cases)
9 -
10 - ### Environment Setup
11 -
12 - - [ ] PostgreSQL running locally with migrations applied (`cargo sqlx migrate run`)
13 - - [ ] Server running (`cargo run` or release binary)
14 - - [ ] `.env` has Stripe **test** keys (sk_test_*, not sk_live_*)
15 - - [ ] `.env` has SIGNING_SECRET set
16 - - [ ] S3 credentials configured (Hetzner Object Storage or compatible)
17 - - [ ] Postmark token set, or accept console-logged emails for dev
18 - - [ ] ADMIN_USER_ID set to your user's UUID
19 -
20 - ### Tips
21 -
22 - - Open browser devtools Network tab — HTMX requests show as XHR, check for 422/500s
23 - - Run server in a second terminal so you can watch logs in real time
24 - - Use incognito/private window when testing auth flows to avoid session bleed
25 - - Stripe test card: `4242 4242 4242 4242`, any future expiry, any CVC
26 -
27 - ---
28 -
29 - ## P0 — Critical Path
30 -
31 - > If any of these fail, do not launch.
32 -
33 - ### Signup → Verify → Login → Logout
34 -
35 - Tested against `testaccount123` (`test@makenot.work`) on 2026-05-16.
36 -
37 - - [x] `GET /join` — signup form renders
38 - - [x] Submit signup with valid username, email, password (8+ chars)
39 - - [x] Server logs verification email (or Postmark sends it) — implied by successful click below
40 - - [x] Verification link in email works (`/verify-email?user=...&expires=...&sig=...`)
41 - - [x] After verification, email_verified flag is true — confirmed in prod DB 2026-05-16 21:05 UTC
42 - - [x] `GET /login` — login form renders
43 - - [x] Login with correct credentials — redirects to `/dashboard`
44 - - [x] `POST /logout` — session destroyed, redirects to `/`
45 - - [x] Accessing `/dashboard` after logout redirects to `/login`
46 - - [x] Login with wrong password — shows error, does not reveal whether user exists (prod logs confirm `Failed login attempt` warn with `attempts` counter — no user enumeration in response)
47 - - [x] Resend verification email works (`/api/resend-verification`) — confirmed during testaccount123 signup flow
48 -
49 - ### Account Lockout + Recovery
50 -
51 - Tested against `testaccount123` on 2026-05-16 21:09 UTC.
52 -
53 - - [x] Fail login 5 times — account locks for 15 minutes (DB: `failed_login_attempts=5`, `locked_until=21:24:54`)
54 - - [x] Lockout notification email sent with one-time login link (Postmark log: subject "Security alert: Account locked")
55 - - [x] One-time login link works (logs you in) — DB cleared to `attempts=0, locked_until=NULL` after click
56 - - [x] One-time login link cannot be reused (single-use) — second click on same link rejected
57 - - [ ] After lockout expires, normal login works again — N/A (lockout cleared by OTP, not by timer); covered by normal login already verified above
58 -
59 - ### Password Reset
60 -
61 - Tested against `testaccount123` on 2026-05-16 21:13 UTC.
62 -
63 - - [x] `GET /forgot-password` — form renders
64 - - [x] Submit email — reset email sent (15-minute expiry link)
65 - - [x] Reset link loads form (`/reset-password?user=...&expires=...&sig=...`)
66 - - [x] Submit new password — succeeds, can login with new password (breached-password advisory fired non-blocking, breach count 2,557 — working as designed)
67 - - [x] Old password no longer works
68 - - [x] Expired reset link rejected
69 - - [x] Reusing same reset link after password change rejected (HMAC includes password hash)
70 -
71 - ### Creator Onboarding
72 -
73 - - [ ] As a regular user, `/dashboard` creator tab shows waitlist apply form
74 - - [ ] Submit waitlist application with pitch text
75 - - [ ] As admin, `GET /admin/waitlist` — shows pending entries
76 - - [ ] Approve entry via `POST /api/admin/waitlist/{id}/approve`
77 - - [ ] Approved user now has can_create_projects flag
78 - - [ ] Approved user sees Stripe Connect setup in dashboard
79 - - [ ] `GET /stripe/connect` — disclaimer page renders
80 - - [ ] `POST /stripe/connect/proceed` — redirects to Stripe OAuth (use test mode)
81 - - [ ] Stripe callback (`/stripe/callback`) saves account ID
82 - - [ ] Dashboard now shows connected Stripe status
83 -
84 - ### Content Creation + Publishing
85 -
86 - - [ ] Create project (`POST /api/projects`) — appears in dashboard
87 - - [ ] Project page renders at `/p/{slug}`
88 - - [ ] Create text item — set title, price (free), description
89 - - [ ] Edit text body (`PUT /api/items/{id}/text`) — markdown renders correctly
90 - - [ ] Create audio item — presign upload, upload file to S3, confirm
91 - - [ ] Audio player works on item page (`/i/{item_id}`)
92 - - [ ] Create download item — presign version upload, upload, confirm
93 - - [ ] Download link works for authorized users
94 - - [ ] Set item visibility to public — appears on `/discover`
95 - - [ ] Set item visibility to private — disappears from `/discover`
96 - - [ ] Create paid item (set price > 0)
97 -
98 - ### Purchase Flow (Fixed Price)
99 -
100 - - [ ] As buyer (different account), browse `/discover` — find the paid item
101 - - [ ] `GET /purchase/{item_id}` — purchase page shows price and fee breakdown
102 - - [ ] `POST /stripe/checkout/{item_id}` — redirects to Stripe Checkout
103 - - [ ] Complete payment with test card (`4242 4242 4242 4242`)
104 - - [ ] `/stripe/success` — success page renders
105 - - [ ] Webhook fires (`checkout.session.completed`) — transaction recorded
106 - - [ ] Item appears in buyer's `/library`
107 - - [ ] Buyer can access item content (stream audio, read text, download file)
108 - - [ ] Cancel checkout — `/stripe/cancel` renders, no transaction created
109 -
110 - ### Pay-What-You-Want (PWYW) Purchase
111 -
112 - - [ ] Create PWYW item with $0 minimum — save succeeds
113 - - [ ] Purchase page shows PWYW input with suggested prices
114 - - [ ] Complete purchase at $0 — item added to library, no Stripe checkout
115 - - [ ] Complete purchase at custom amount (e.g. $5) — Stripe Checkout, item in library
116 - - [ ] Create PWYW item with non-zero minimum (e.g. $5)
117 - - [ ] Attempt purchase below minimum — rejected
118 -
119 - ### Subscription Flow
120 -
121 - - [ ] Create subscription tier on a project (e.g. $3/mo)
122 - - [ ] As buyer, subscription page renders with tier details
123 - - [ ] `POST /stripe/subscribe/{project_id}` — redirects to Stripe Checkout (subscription mode)
124 - - [ ] Complete subscription with test card
125 - - [ ] Webhook fires (`customer.subscription.created`) — subscription recorded
126 - - [ ] Subscriber can access subscriber-only items
127 - - [ ] Non-subscriber cannot access subscriber-only content
128 - - [ ] Cancel subscription — access continues until end of billing period
129 -
130 - ### Discount Codes
131 -
132 - - [ ] Create discount code (e.g. LAUNCH50, 50% off, limited uses)
133 - - [ ] Apply code at checkout — price reduced correctly
134 - - [ ] Discount shows in fee breakdown
135 - - [ ] Exhausted code rejected (after max uses reached)
136 - - [ ] Expired code rejected
137 -
138 - ### License Keys
139 -
140 - - [ ] Create item with license keys enabled
141 - - [ ] After purchase, license key displayed to buyer
142 - - [ ] `POST /api/licenses/{key}/activate` — activation succeeds
143 - - [ ] Activation count increments
144 - - [ ] `GET /api/licenses/{key}/verify` — returns valid status
145 - - [ ] Exceed activation limit — activation rejected
146 -
147 - ### Free Item Claim
148 -
149 - Tested by `max` on GO (GoingsOn Desktop free item) — transaction recorded 2026-05-10.
150 -
151 - - [x] As buyer, find a free item on `/discover`
152 - - [x] `POST /api/library/add/{item_id}` — item added to library (transaction row, status=completed, amount=0)
153 - - [x] Item content accessible
154 - - [x] `DELETE /api/library/remove/{item_id}` — item removed from library
155 -
156 - ### File Upload + Delivery
157 -
158 - - [ ] Presign request (`POST /api/upload/presign`) returns valid S3 URL
159 - - [ ] Direct upload to presigned URL succeeds
160 - - [ ] Confirm upload (`POST /api/upload/confirm`) stores S3 key
161 - - [ ] Audio streaming URL (`GET /api/stream/{item_id}`) returns presigned URL
162 - - [ ] Version file download (`GET /api/versions/{version_id}/download`) works
163 - - [ ] Cover image upload and display works
164 - - [ ] Presigned URLs expire (check after 1+ hours)
165 -
166 - ---
167 -
168 - ## P1 — Core Features
169 -
170 - ### Dashboard
171 -
172 - - [ ] `/dashboard` renders with projects list
173 - - [ ] Details tab (`/dashboard/tabs/details`) — shows username, email, bio
174 - - [ ] Payments tab (`/dashboard/tabs/payments`) — shows transaction history
175 - - [ ] Projects tab (`/dashboard/tabs/projects`) — lists all projects
176 - - [ ] Creator tab (`/dashboard/tabs/creator`) — shows waitlist or Stripe status
177 - - [ ] Profile update (`PUT /api/users/me`) — display name and bio save correctly
178 - - [ ] Password update (`PUT /api/users/me/password`) — works with correct current password
179 -
180 - ### Project Management
181 -
182 - - [ ] Project dashboard (`/dashboard/project/{slug}`) renders
183 - - [ ] Overview tab — project stats display
184 - - [ ] Content tab — items listed
185 - - [ ] Analytics tab — renders (even if empty)
186 - - [ ] Settings tab — project settings editable
187 - - [ ] Update project (`PUT /api/projects/{id}`) — title, description, type, visibility
188 - - [ ] Delete project (`DELETE /api/projects/{id}`) — cascade deletes items
189 -
190 - ### Item Management
191 -
192 - - [ ] Item dashboard (`/dashboard/item/{id}`) renders
193 - - [ ] Inline edit row (`/dashboard/item/{id}/edit-row`) works via HTMX
194 - - [ ] Update item metadata (`PUT /api/items/{id}`) — title, price, type, description
195 - - [ ] Version list (`GET /api/items/{id}/versions`) renders
196 - - [ ] Create new version (`POST /api/items/{id}/versions`) with file upload
197 -
198 - ### Discover
199 -
200 - - [ ] `/discover` renders with default results
201 - - [ ] Search by text — results filter correctly (trigram search)
202 - - [ ] Filter by category — correct items shown, category counts update
203 - - [ ] Filter by price range (Free, <$25, $25-50, $50-100, $100+)
204 - - [ ] Switch between Items and Projects mode
205 - - [ ] Sort options work (newest, oldest, price, sales)
206 - - [ ] Pagination — next/prev pages load via HTMX
207 - - [ ] `/discover/results` partial loads correctly (check Network tab)
208 -
209 - ### Public Profiles
210 -
211 - - [ ] `/u/{username}` — user profile renders with projects and custom links
212 - - [ ] `/p/{slug}` — project page renders with items
213 - - [ ] `/i/{item_id}` — text item renders markdown correctly
214 - - [ ] `/i/{item_id}` — audio item shows player with chapters
215 - - [ ] `/i/{item_id}` — download item shows version list
216 -
217 - ### Custom Links
218 -
219 - - [ ] Create link (`POST /api/links`) — appears on profile
220 - - [ ] Update link (`PUT /api/links/{id}`) — changes reflected
221 - - [ ] Delete link (`DELETE /api/links/{id}`) — removed from profile
222 - - [ ] Reorder links (`PUT /api/links/reorder`) — order persists
223 -
224 - ### Tags + Chapters
225 -
226 - - [ ] Add tag to item (`POST /api/items/{id}/tags`) — tag appears
227 - - [ ] Remove tag (`DELETE /api/items/{id}/tags/{tag}`) — tag removed
228 - - [ ] Create chapter (`POST /api/items/{id}/chapters`) — chapter marker appears
229 - - [ ] Update chapter (`PUT /api/chapters/{id}`) — changes saved
230 - - [ ] Delete chapter (`DELETE /api/chapters/{id}`) — removed
231 - - [ ] Chapters display on audio item page with correct timestamps
232 -
233 - ### RSS Feeds
234 -
235 - - [ ] `/u/{username}/rss` — valid RSS 2.0, includes public items
236 - - [ ] `/p/{slug}/rss` — valid RSS 2.0, includes project's public items
237 - - [ ] Feed updates when new item published
238 -
239 - ### Blog Posts
240 -
241 - - [ ] Create blog post on a project — title, slug, body (markdown)
242 - - [ ] Blog post renders at `/p/{slug}/blog/{post_slug}`
243 - - [ ] Blog post appears in project RSS feed
244 - - [ ] Edit blog post — changes saved and visible
245 - - [ ] Delete blog post — removed from project page and RSS
246 -
247 - ### Two-Factor Authentication
248 -
249 - - [ ] Enable TOTP 2FA — QR code and secret displayed
250 - - [ ] Login with 2FA enabled — prompted for TOTP code after password
251 - - [ ] Correct TOTP code — login succeeds
252 - - [ ] Wrong TOTP code — login rejected
253 - - [ ] Backup codes — one works, same code cannot be reused
254 - - [ ] Disable 2FA — login no longer prompts for code
255 -
256 - ### Passkeys (WebAuthn)
257 -
258 - - [ ] Register passkey from dashboard security section
259 - - [ ] Login with passkey — bypasses password
260 - - [ ] Remove passkey — can no longer use it to login
261 -
262 - ### Git Browser
263 -
264 - - [ ] `/git/{username}/{repo}` — file tree renders
265 - - [ ] Click file — blob view with syntax highlighting
266 - - [ ] `/git/{username}/{repo}/commits` — commit log renders
267 - - [ ] Click commit — diff view renders
268 - - [ ] `/git/{username}/{repo}/blame/{path}` — blame view renders
269 - - [ ] Clone URL displayed and correct (`ssh.makenot.work`)
270 -
271 - ### Data Export
272 -
273 - - [ ] Projects export (`POST /api/export/projects`) — downloads JSON
274 - - [ ] Sales export (`POST /api/export/sales`) — downloads CSV
275 - - [ ] Purchases export (`POST /api/export/purchases`) — downloads CSV
276 - - [ ] Exported data is accurate (spot-check a few records)
277 -
278 - ---
279 -
280 - ## P2 — Edge Cases + Security
281 -
282 - ### Access Control
283 -
284 - - [ ] Cannot view another user's dashboard (`/dashboard` only shows your data)
285 - - [ ] Cannot edit another user's project (`PUT /api/projects/{id}` — 403/404)
286 - - [ ] Cannot delete another user's item (`DELETE /api/items/{id}` — 403/404)
287 - - [ ] Cannot access paid item content without purchase
288 - - [ ] Cannot access private/draft items via direct URL
289 - - [ ] Admin routes (`/admin/*`) return 403 for non-admin users
290 - - [ ] Stripe disconnect (`DELETE /api/users/me/stripe`) only affects your account
291 -
292 - ### Rate Limiting
293 -
294 - - [ ] Hit `/login` rapidly (>5 times) — returns 429 Too Many Requests
295 - - [ ] Hit `/api/upload/presign` rapidly (>10 times) — returns 429
296 - - [ ] Hit `/api/export/projects` rapidly (>3 times) — returns 429
297 - - [ ] Rate limits reset after the window passes
298 -
299 - ### CSRF
300 -
301 - - [ ] Submit a POST/PUT/DELETE without CSRF token — rejected
302 - - [ ] Submit with invalid CSRF token — rejected
303 - - [ ] Normal form submissions with valid token — succeed
304 - - [ ] Exempt routes work without CSRF: `/login`, `/join`, `/logout`, `/stripe/webhook`
305 -
306 - ### Input Validation
307 -
308 - - [ ] XSS attempt in username/bio/project fields — HTML escaped in output
309 - - [ ] SQL injection attempt in search/form fields — no errors, input treated as text
310 - - [ ] Overlong input (10k+ chars in text fields) — rejected or truncated gracefully
311 - - [ ] Negative price on item — rejected
312 - - [ ] Zero-length required fields — rejected with validation error
313 - - [ ] Markdown rendering sanitized (no script tags, no raw HTML that could execute)
314 -
315 - ### Account Deletion
316 -
317 - - [ ] Request deletion (`POST /api/account/request-deletion`) — confirmation email sent
318 - - [ ] Confirmation link (`/confirm-delete?user=...&expires=...&sig=...`) — deletes account
319 - - [ ] After deletion, login with old credentials fails
320 - - [ ] Deleted user's public pages return 404
321 - - [ ] Purchases by deleted user are preserved (preserve_purchases migration)
322 -
323 - ### Error Pages
324 -
325 - - [ ] Hit nonexistent route — custom 404 page renders
326 - - [ ] Error templates render correctly (check `/deploy/error-pages/`)
327 -
328 - ---
329 -
330 - ## Infrastructure Verification
331 -
332 - > Run these checks on the production server after deploy.
333 -
334 - ### DNS + HTTPS
335 -
336 - - [ ] A record points to server IP (`dig makenot.work`)
337 - - [ ] HTTPS certificate valid (Cloudflare Origin CA, 15yr wildcard)
338 - - [ ] `Strict-Transport-Security` header present
339 - - [ ] `http://makenot.work` redirects to `https://makenot.work`
340 - - [ ] `www.makenot.work` redirects to `makenot.work` (if configured)
341 -
342 - ### Security Headers
343 -
344 - - [ ] `Content-Security-Policy` header present
345 - - [ ] `X-Frame-Options: DENY` or `SAMEORIGIN`
346 - - [ ] `X-Content-Type-Options: nosniff`
347 - - [ ] `Referrer-Policy` header present
348 - - [ ] `Permissions-Policy` header present
349 - - [ ] Check headers: `curl -I https://makenot.work`
350 -
351 - ### Systemd Service
352 -
353 - - [ ] Service running: `systemctl status makenotwork`
354 - - [ ] Restart policy active: `Restart=on-failure` in service file
355 - - [ ] Service starts on boot: `systemctl is-enabled makenotwork`
356 - - [ ] Security hardening active (check `ProtectSystem`, `NoNewPrivileges`, etc. in service file)
357 - - [ ] Test restart: `systemctl restart makenotwork` — comes back healthy
358 -
359 - ### Database
360 -
361 - - [ ] Migrations applied: all 45 migrations (`cargo sqlx migrate info` or check schema)
362 - - [ ] Connection healthy: `GET /health` shows database green
363 - - [ ] Demo seed data removed (migrations 011-014, 016-017 are seed data — verify no test users/items in production)
364 - - [ ] pg_trgm extension installed (required for search)
365 -
366 - ### Backups
367 -
368 - - [ ] Cron job configured: `crontab -l` shows daily 3 AM backup
369 - - [ ] Manual backup works: `bash deploy/backup-db.sh`
370 - - [ ] Backup file created and non-empty in backup directory
371 - - [ ] Test restore to a scratch database (see `RECOVERY.md`)
372 - - [ ] 30-day retention — old backups cleaned up
373 -
374 - ### Environment
375 -
376 - - [ ] `.env` file permissions: `600` (owner read/write only)
377 - - [ ] No test keys in production (grep for `sk_test_`, `pk_test_`)
378 - - [ ] `SIGNING_SECRET` is set and is a strong random value
379 - - [ ] `HOST_URL` is `https://makenot.work` (not localhost)
380 - - [ ] `STRIPE_WEBHOOK_SECRET` matches the webhook configured in Stripe dashboard
381 - - [ ] `POSTMARK_TOKEN` is set (not console mode)
382 - - [ ] `ADMIN_USER_ID` is set to the correct UUID
383 -
384 - ### Health Endpoint
385 -
386 - - [ ] `GET /health` returns 200
387 - - [ ] Database: connected, shows table counts
388 - - [ ] Sessions: store active
389 - - [ ] S3: configured
390 - - [ ] Stripe: configured, **live mode** (not test)
391 - - [ ] Email: Postmark (not console)
392 -
393 - ### Logs
394 -
395 - - [ ] Server logs flowing: `journalctl -u makenotwork -f`
396 - - [ ] Caddy logs flowing: `journalctl -u caddy -f`
397 - - [ ] No errors or panics on startup
398 - - [ ] A test request shows up in logs
399 -
400 - ### Firewall
401 -
402 - - [ ] Ports 80, 443 open to all (required for custom domains + on-demand TLS): `ufw status`
403 - - [ ] Port 22 open (SSH)
404 - - [ ] All other ports blocked
405 - - [ ] makenot.work protected by Cloudflare mTLS even with open ports
406 -
407 - ---
408 -
409 - ## Sign-Off
410 -
411 - | Field | Value |
412 - |-------|-------|
413 - | Date | |
414 - | Tester | |
415 - | Environment | local / staging / production |
416 - | Automated tests passing | yes / no |
417 - | P0 result | pass / fail |
418 - | P1 result | pass / fail |
419 - | P2 result | pass / fail / skipped |
420 - | Infrastructure result | pass / fail / N/A |
421 - | Notes | |