max / makenotwork

Add server docs, rollback guides, slim CLAUDE.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Max J. <87768334+MaxJMath@users.noreply.github.com> · 2026-04-16 02:19 UTC

Commit: c4a3178d009e954fcd1403bb39b3fcf10c8ccf8c

Parent: b4c4cf8

7 files changed, +1202 insertions, -174 deletions

M CLAUDE.md +18 -174

			@@ -30,47 +30,7 @@ Fair creator platform with 0% platform fee (only Stripe's ~3% processing fee). M
30	30
31	31		## Ecosystem
32	32
33		-	This is a monorepo containing the MNW server and all related ecosystem projects:
34		-
35		-	\| Project \| Path \| Description \|
36		-	\|---------\|------\|-------------\|
37		-	\| MNW Server \| `server/` \| Rust/Axum backend, HTMX frontend, PostgreSQL, Stripe Connect \|
38		-	\| Multithreaded \| `multithreaded/` \| Forum software integrated with MNW (Rust/Axum/PostgreSQL, MNW OAuth) \|
39		-	\| PoM \| `pom/` \| Production operations monitor (health checks, TLS tracking, email alerts) \|
40		-	\| mnw-cli \| `mnw-cli/` \| CLI tool for MNW platform \|
41		-
42		-	Shared libraries live at `shared/` (docengine, tagtree, synckit-client, theme-common, s3-storage, themes, tauri-updater-ui).
43		-
44		-	## Repository Layout
45		-
46		-	```
47		-	MNW/ # Monorepo root
48		-	server/ # MNW server (crate root)
49		-	src/ # Application source
50		-	migrations/ # SQLx migrations (numbered, auto-applied on boot)
51		-	templates/ # Askama HTML templates
52		-	static/ # CSS, JS, fonts, images
53		-	tests/ # Integration tests (workflows/, load/, harness/)
54		-	deploy/ # Deployment scripts and config files
55		-	deploy.sh # Cross-compile + upload + restart
56		-	makenotwork.service # systemd unit file
57		-	Caddyfile # Reverse proxy config
58		-	backup-db.sh # DB backup script
59		-	error-pages/ # Custom 404/500/502 pages
60		-	site-docs/ # DocEngine content (public/ and unpublished/)
61		-	docs/ # Server-specific docs (todo, audit, architecture, etc.)
62		-	multithreaded/ # Forum software
63		-	pom/ # Production operations monitor
64		-	mnw-cli/ # CLI tool
65		-	shared/ # Shared libraries
66		-	docengine/ # Markdown rendering + documentation engine
67		-	tagtree/ # Hierarchical tag standard
68		-	synckit-client/ # SyncKit cloud sync client SDK
69		-	theme-common/ # Theme loading + parsing
70		-	s3-storage/ # S3-compatible storage abstraction
71		-	themes/ # TOML theme definitions
72		-	tauri-updater-ui/ # OTA update UI components
73		-	```
	33	+	Monorepo: `server/`, `multithreaded/`, `pom/`, `mnw-cli/`, `shared/`. No root `[workspace]` — each project is its own crate (see `server/docs/architecture.md` for full layout and why).
74	34
75	35		## Coding Patterns
76	36
			@@ -82,147 +42,31 @@ See `multithreaded/CONTRIBUTING.md` for MT-specific patterns (MNW OAuth, shared
82	42
83	43		## Versioning
84	44
85		-	- Semver in `Cargo.toml` (`env!("CARGO_PKG_VERSION")` compiles it into the binary for Sentry release strings)
	45	+	- Semver in `Cargo.toml` (`env!("CARGO_PKG_VERSION")` compiles it into the binary)
86	46		- Before every deploy to production, ask the user what version to set — never auto-bump
87	47		- Version bump = edit `Cargo.toml` version field before building
88	48
89		-	---
	49	+	## SyncKit
90	50
91		-	## MNW SyncKit
	51	+	E2E encrypted cloud sync + OTA updates for indie apps. Design philosophy: general-purpose first, E2E encrypted by default, bring your own schema, auth via MNW accounts. See `server/docs/architecture.md` for component locations and `shared/synckit-client/` crate docs for the SDK.
92	52
93		-	Developer infrastructure for indie apps, hosted on Makenotwork.
	53	+	## Operations Quick Reference
94	54
95		-	### Services
	55	+	Detailed infrastructure, deployment, and troubleshooting docs live in `server/docs/`:
	56	+	- Deployment: `server/deploy/deploy.sh` (run from `MNW/server/`)
	57	+	- Rollback: `server/docs/rollback.md`
	58	+	- Troubleshooting: `server/docs/troubleshooting.md`
	59	+	- Incident response: `../_meta/incident_response.md`
	60	+	- Infrastructure diagrams: `../_meta/diagrams/infra/index.md`
	61	+	- Service accounts: `../_meta/service_accounts.md`
	62	+	- Testing: `server/docs/test_plan.md`
	63	+	- Schema: `server/docs/schema.md`
96	64
97		-	- Cloud Sync — Push/pull changelog sync with E2E encryption, device management, conflict resolution
98		-	- OTA Updates — App auto-update server (Tauri-compatible protocol), no app store dependency
	65	+	### Key IPs (for LLM-assisted SSH/deploy commands)
99	66
100		-	### Design Philosophy
101		-
102		-	- General-purpose first — API and SDK decisions should make sense for any app, not just GO
103		-	- E2E encrypted by default — Server stores only encrypted blobs, never plaintext user data
104		-	- Bring your own schema — Table names, row IDs, and data shapes are opaque to the server
105		-	- Auth via MNW accounts — Users authenticate with their Makenot.work credentials
106		-
107		-	### Components
108		-
109		-	\| Component \| Location \| Role \|
110		-	\|-----------\|----------\|------\|
111		-	\| Server API \| `server/src/routes/synckit.rs` \| Axum endpoints (auth, push/pull, devices, keys) \|
112		-	\| Server DB \| `server/src/db/synckit.rs` \| PostgreSQL queries (sync_apps, sync_devices, sync_log, sync_keys) \|
113		-	\| Server Auth \| `server/src/synckit_auth.rs` \| JWT token creation + extraction \|
114		-	\| Client SDK \| `shared/synckit-client/` \| Rust crate — HTTP client, E2E crypto, keychain storage \|
115		-	\| Integration tests \| `server/tests/workflows/synckit.rs` \| 7 tests covering auth, devices, push/pull, keys, validation \|
116		-
117		-	### Consumers
118		-
119		-	\| App \| What it syncs \| Status \|
120		-	\|-----\|---------------\|--------\|
121		-	\| GoingsOn \| Tasks, projects, events, contacts, emails \| Implemented \|
122		-	\| Balanced Breakfast \| Configs, feed sources, plugin manifests \| Integrated \|
123		-	\| audiofiles \| Sample metadata, tags, VFS mappings \| Integrated \|
124		-
125		-	---
	67	+	- Hetzner (prod): `100.120.174.96` (Tailscale) / `5.78.144.244` (public)
	68	+	- Astra (dev/test): `100.106.221.39` (Tailscale)
126	69
127	70		## CI
128	71
129		-	MNW CI runs self-hosted on astra (`server/deploy/run-ci.sh` — check, test, clippy, audit). GO, BB, and AF still have `.build.yml` manifests for builds.sr.ht (Arch Linux + Rust). Sourcehut (`https://sr.ht/~maxmj/`) remains active as a git mirror. MNW has a built-in git browser (G1, `git2`-based) that reads bare repos from disk.
130		-
131		-	## Infrastructure Diagrams
132		-
133		-	Mermaid diagrams documenting the full MNW ecosystem live at `../_meta/diagrams/infra/`. See `index.md` for the table of contents, or open `viewer.html` in a browser to view all 52 diagrams rendered. Keep diagrams in sync when making infrastructure changes.
134		-
135		-	## Production Server (hetzner)
136		-
137		-	Hetzner VPS, x86_64 Linux. Tailscale hostname: `alpha-west-1` (IP: `100.120.174.96`). Public IP: `5.78.144.244`. SSH as `root@100.120.174.96` (via Tailscale).
138		-
139		-	### Filesystem
140		-
141		-	```
142		-	/opt/makenotwork/
143		-	makenotwork # Application binary
144		-	.env # Environment variables (secrets)
145		-	static/ # CSS, JS, fonts, images
146		-	error-pages/ # Custom error pages served by Caddy
147		-	backup-db.sh # Database backup script
148		-	src/ # Deployed source (for sqlx migrations)
149		-	docs/ # Deployed docs (for /docs pages)
150		-
151		-	/opt/git/ # Bare git repos for source browser
152		-	makenotwork.git/
153		-	synckit-client.git/
154		-	...
155		-
156		-	/etc/caddy/Caddyfile # Caddy reverse proxy config
157		-	/etc/systemd/system/makenotwork.service # systemd unit
158		-	```
159		-
160		-	### Services
161		-
162		-	- `makenotwork` — the application (systemd, runs as `makenotwork` user)
163		-	- `caddy` — reverse proxy + TLS
164		-	- `postgresql` — database (`makenotwork` db, `makenotwork` user)
165		-
166		-	### Deployment
167		-
168		-	From the `MNW/server/` directory:
169		-	```sh
170		-	./deploy/deploy.sh # Full: build + config + binary + restart
171		-	./deploy/deploy.sh --quick # Build + binary + restart (no config upload)
172		-	./deploy/deploy.sh --config # Config files only (Caddyfile, systemd, error pages, static)
173		-	```
174		-
175		-	Cross-compiles with `cargo zigbuild` (requires `zig`, `cargo-zigbuild`, `x86_64-unknown-linux-gnu` target).
176		-
177		-	## Astra (dev/test server)
178		-
179		-	General-purpose Linux box on Tailscale. SSH as `max@100.106.221.39`.
180		-
181		-	- OS: Pop!_OS 24.04 LTS, aarch64 (96 cores, 125GB RAM, 929GB NVMe)
182		-	- Tailscale hostname: `astra` (IP: `100.106.221.39`)
183		-	- PostgreSQL 16 — tuned: `max_connections=200`, `shared_buffers=4GB`, `work_mem=64MB`, `maintenance_work_mem=512MB`
184		-	- PG roles: `postgres` (super), `max` (local, createdb), `mnw_staging` (createdb)
185		-	- Rust 1.94 — installed via rustup (`~/.cargo/bin/cargo`)
186		-	- Databases: `postgres`, `devdb`, `makenotwork_staging`
187		-	- Staging copy: `/home/max/staging/makenotwork/` (not a git repo, deployed copy)
188		-	- Environment: `RUST_TEST_THREADS=8` (in `.bashrc` and `.profile`), `ulimit -n 65536` (in `/etc/security/limits.conf`)
189		-
190		-	### Running integration tests on astra
191		-
192		-	```sh
193		-	# Using the test runner script (handles env vars and orphan cleanup):
194		-	ssh 100.106.221.39
195		-	/home/max/staging/run-tests.sh # Run all
196		-	/home/max/staging/run-tests.sh auth:: # Run filtered
197		-
198		-	# Or manually:
199		-	cd /home/max/staging/makenotwork
200		-	TEST_DATABASE_URL="postgres:///postgres" cargo test --test integration -- --test-threads=8
201		-	```
202		-
203		-	Important: Use `--test-threads=8` (or similar). The default parallelism (96 on astra) overwhelms PostgreSQL with too many simultaneous `CREATE DATABASE` calls. The `RUST_TEST_THREADS=8` env var handles this automatically.
204		-
205		-	### Cleaning up orphaned test databases
206		-
207		-	Test databases (`mnw_test_*`) are orphaned if the test process is killed. The `run-tests.sh` script cleans these up automatically. Manual cleanup:
208		-	```sh
209		-	psql -t -c "SELECT datname FROM pg_database WHERE datname LIKE 'mnw_test_%';" postgres \
210		-	\| xargs -I{} psql -c "DROP DATABASE IF EXISTS \"{}\";" postgres
211		-	```
212		-
213		-	## External Services
214		-
215		-	- Stripe Connect — payments (live mode)
216		-	- Hetzner Object Storage — S3-compatible file storage (fsn1 region)
217		-	- Postmark — transactional email (password reset, verification, purchase receipts, notifications). Live mode.
218		-	- Cloudflare — DNS, CDN, DDoS protection
219		-	- Fastmail — business email (support@, legal@, max@)
220		-
221		-	## Testing
222		-
223		-	```sh
224		-	cargo test # Unit tests (no DB needed)
225		-
226		-	# Integration tests (need a running Postgres):
227		-	TEST_DATABASE_URL="postgres://user:pass@host:5432/postgres" cargo test --test integration
228		-	```
	72	+	MNW CI runs self-hosted on astra (`server/deploy/run-ci.sh`). MNW has a built-in git browser (G1, `git2`-based) at `/source/`.

A multithreaded/deploy/ROLLBACK.md +146

		@@ -0,0 +1,146 @@
1	+	# Rollback Guide — Multithreaded
2	+
3	+	## Quick Rollback (Re-deploy Previous Binary)
4	+
5	+	MT is deployed to two targets: Hetzner (production) and Astra (staging). Rollback means re-building a previous commit.
6	+
7	+	### Steps (Hetzner — production)
8	+
9	+	1. Identify the last known-good commit:
10	+	```bash
11	+	cd MNW/multithreaded
12	+	git log --oneline -10
13	+	```
14	+
15	+	2. Check out and build the previous version:
16	+	```bash
17	+	git checkout <commit-hash>
18	+	cargo zigbuild --release --target x86_64-unknown-linux-gnu
19	+	```
20	+
21	+	3. Deploy the rollback binary:
22	+	```bash
23	+	ssh root@100.120.174.96 "systemctl stop multithreaded \|\| true"
24	+	scp target/x86_64-unknown-linux-gnu/release/multithreaded root@100.120.174.96:/opt/multithreaded/multithreaded
25	+	ssh root@100.120.174.96 "chmod +x /opt/multithreaded/multithreaded && chown multithreaded:multithreaded /opt/multithreaded/multithreaded"
26	+	ssh root@100.120.174.96 "systemctl start multithreaded"
27	+	```
28	+
29	+	4. Verify:
30	+	```bash
31	+	ssh root@100.120.174.96 "systemctl status multithreaded --no-pager"
32	+	ssh root@100.120.174.96 "curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:3400"
33	+	```
34	+
35	+	5. Return to main branch:
36	+	```bash
37	+	git checkout main
38	+	```
39	+
40	+	### Steps (Astra — staging)
41	+
42	+	Astra builds natively (aarch64). The deploy script rsyncs source and builds on Astra.
43	+
44	+	1. Check out the previous commit locally:
45	+	```bash
46	+	git checkout <commit-hash>
47	+	```
48	+
49	+	2. Re-deploy using the normal deploy script:
50	+	```bash
51	+	./deploy/deploy.sh
52	+	```
53	+	This rsyncs the checked-out source to Astra, builds there, and restarts.
54	+
55	+	3. Return to main branch:
56	+	```bash
57	+	git checkout main
58	+	```
59	+
60	+	## Emergency Stop
61	+
62	+	```bash
63	+	# Hetzner (production)
64	+	ssh root@100.120.174.96 "systemctl stop multithreaded"
65	+
66	+	# Astra (staging)
67	+	ssh max@100.106.221.39 "sudo systemctl stop multithreaded"
68	+	```
69	+
70	+	To restart:
71	+	```bash
72	+	ssh root@100.120.174.96 "systemctl start multithreaded"
73	+	ssh max@100.106.221.39 "sudo systemctl start multithreaded"
74	+	```
75	+
76	+	## Database Restore
77	+
78	+	MT does not have automated backups. For a manual backup/restore:
79	+
80	+	### Create a manual backup
81	+
82	+	```bash
83	+	# Hetzner
84	+	ssh root@100.120.174.96 "sudo -u multithreaded pg_dump multithreaded \| gzip > /opt/multithreaded/mt-backup-$(date +%Y%m%d).sql.gz"
85	+
86	+	# Astra
87	+	ssh max@100.106.221.39 "pg_dump multithreaded \| gzip > /tmp/mt-backup-$(date +%Y%m%d).sql.gz"
88	+	```
89	+
90	+	### Restore from backup
91	+
92	+	1. Stop the application:
93	+	```bash
94	+	ssh root@100.120.174.96 "systemctl stop multithreaded"
95	+	```
96	+
97	+	2. Back up current state:
98	+	```bash
99	+	ssh root@100.120.174.96 "sudo -u multithreaded pg_dump multithreaded \| gzip > /opt/multithreaded/mt-pre-restore.sql.gz"
100	+	```
101	+
102	+	3. Drop and recreate:
103	+	```bash
104	+	ssh root@100.120.174.96 "sudo -u postgres psql -c 'DROP DATABASE multithreaded;'"
105	+	ssh root@100.120.174.96 "sudo -u postgres psql -c \"CREATE DATABASE multithreaded OWNER multithreaded;\""
106	+	```
107	+
108	+	4. Restore:
109	+	```bash
110	+	ssh root@100.120.174.96 "gunzip -c /opt/multithreaded/mt-backup-YYYYMMDD.sql.gz \| sudo -u multithreaded psql multithreaded"
111	+	```
112	+
113	+	5. Restart (migrations auto-apply on boot):
114	+	```bash
115	+	ssh root@100.120.174.96 "systemctl start multithreaded"
116	+	```
117	+
118	+	## Service Architecture Reference
119	+
120	+	### Hetzner (production)
121	+
122	+	- Binary: `/opt/multithreaded/multithreaded`
123	+	- Config: `/opt/multithreaded/.env`
124	+	- Static: `/opt/multithreaded/static/`
125	+	- Migrations: `/opt/multithreaded/migrations/`
126	+	- Systemd unit: `/etc/systemd/system/multithreaded.service`
127	+	- Logs: `journalctl -u multithreaded -f`
128	+	- Port: 127.0.0.1:3400 (Caddy reverse proxies `forums.makenot.work`)
129	+	- DB: PostgreSQL `multithreaded` database, `multithreaded` user (peer auth)
130	+	- Domain: `forums.makenot.work` (Cloudflare-proxied)
131	+
132	+	### Astra (staging)
133	+
134	+	- Binary: `/opt/multithreaded/multithreaded`
135	+	- Config: `/opt/multithreaded/.env`
136	+	- Source: `~/src/multithreaded/` (rsynced from local)
137	+	- Shared deps: `~/src/shared/` (docengine, tagtree, s3-storage)
138	+	- Port: 0.0.0.0:3400 (direct access via Tailscale)
139	+	- DB: PostgreSQL `multithreaded` database
140	+
141	+	### Common
142	+
143	+	- Restart policy: `Restart=always`, `RestartSec=5`
144	+	- Depends on: `postgresql.service`
145	+	- Migrations: auto-applied on boot (`sqlx::migrate!()`)
146	+	- Memory limit: 512M (`MemoryMax=512M`)

A server/deploy/ROLLBACK.md +127

		@@ -0,0 +1,127 @@
1	+	# Rollback Guide — MNW Server
2	+
3	+	## Quick Rollback (Re-deploy Previous Binary)
4	+
5	+	The previous binary is overwritten during deploy, so rollback means re-building a previous commit and deploying it.
6	+
7	+	### Steps
8	+
9	+	1. Identify the last known-good commit:
10	+	```bash
11	+	cd MNW/server
12	+	git log --oneline -10
13	+	```
14	+
15	+	2. Check out and build the previous version:
16	+	```bash
17	+	git checkout <commit-hash>
18	+	cargo zigbuild --release --target x86_64-unknown-linux-gnu
19	+	```
20	+
21	+	3. Deploy the rollback binary:
22	+	```bash
23	+	ssh root@100.120.174.96 "systemctl stop makenotwork \|\| true"
24	+	scp target/x86_64-unknown-linux-gnu/release/makenotwork root@100.120.174.96:/opt/makenotwork/makenotwork
25	+	ssh root@100.120.174.96 "chmod +x /opt/makenotwork/makenotwork"
26	+	ssh root@100.120.174.96 "systemctl start makenotwork"
27	+	```
28	+
29	+	4. Verify:
30	+	```bash
31	+	ssh root@100.120.174.96 "systemctl status makenotwork --no-pager"
32	+	ssh root@100.120.174.96 "curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:3000"
33	+	```
34	+
35	+	5. Return to main branch:
36	+	```bash
37	+	git checkout main
38	+	```
39	+
40	+	## Emergency Stop
41	+
42	+	Stop the application immediately. Caddy will serve the 502 error page (auto-retries every 10 seconds).
43	+
44	+	```bash
45	+	ssh root@100.120.174.96 "systemctl stop makenotwork"
46	+	```
47	+
48	+	To restart:
49	+	```bash
50	+	ssh root@100.120.174.96 "systemctl start makenotwork"
51	+	```
52	+
53	+	## Database Restore
54	+
55	+	Backups are created daily at 03:00 UTC by cron (`/opt/makenotwork/backup-db.sh`). Stored at `/opt/makenotwork/backups/`, 30-day retention.
56	+
57	+	### Restore from backup
58	+
59	+	1. Stop the application:
60	+	```bash
61	+	ssh root@100.120.174.96 "systemctl stop makenotwork"
62	+	```
63	+
64	+	2. List available backups:
65	+	```bash
66	+	ssh root@100.120.174.96 "ls -lh /opt/makenotwork/backups/"
67	+	```
68	+
69	+	3. Create a backup of the current (broken) state first:
70	+	```bash
71	+	ssh root@100.120.174.96 "sudo -u makenotwork pg_dump makenotwork \| gzip > /opt/makenotwork/backups/makenotwork-pre-restore.sql.gz"
72	+	```
73	+
74	+	4. Drop and recreate the database:
75	+	```bash
76	+	ssh root@100.120.174.96 "sudo -u postgres psql -c 'DROP DATABASE makenotwork;'"
77	+	ssh root@100.120.174.96 "sudo -u postgres psql -c \"CREATE DATABASE makenotwork OWNER makenotwork;\""
78	+	```
79	+
80	+	5. Restore from backup:
81	+	```bash
82	+	ssh root@100.120.174.96 "gunzip -c /opt/makenotwork/backups/makenotwork-YYYYMMDD-HHMMSS.sql.gz \| sudo -u makenotwork psql makenotwork"
83	+	```
84	+
85	+	6. Restart the application (migrations will run on boot and apply any missing ones):
86	+	```bash
87	+	ssh root@100.120.174.96 "systemctl start makenotwork"
88	+	```
89	+
90	+	7. Verify:
91	+	```bash
92	+	ssh root@100.120.174.96 "systemctl status makenotwork --no-pager"
93	+	ssh root@100.120.174.96 "curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:3000"
94	+	```
95	+
96	+	### Notes on DB Restore
97	+
98	+	- Migrations are auto-applied on boot via sqlx. If you restore an older backup and the current binary has newer migrations, they'll run automatically.
99	+	- If a migration is incompatible with the restored data, you'll need to also rollback the binary (see Quick Rollback above).
100	+	- Data created between the backup and the restore will be lost. There's no WAL archiving — only daily `pg_dump`.
101	+
102	+	## Migration Rollback
103	+
104	+	Migrations are additive (new tables, new columns with defaults). There's no built-in `down` migration. If a migration causes issues:
105	+
106	+	1. Manually write a reversal SQL script
107	+	2. Apply it: `ssh root@100.120.174.96 "sudo -u makenotwork psql makenotwork < /tmp/rollback.sql"`
108	+	3. Delete the migration row from `_sqlx_migrations` so it doesn't conflict:
109	+	```sql
110	+	DELETE FROM _sqlx_migrations WHERE version = <migration_number>;
111	+	```
112	+
113	+	## Service Architecture Reference
114	+
115	+	- Binary: `/opt/makenotwork/makenotwork`
116	+	- Config: `/opt/makenotwork/.env`
117	+	- Static: `/opt/makenotwork/static/`
118	+	- Docs: `/opt/makenotwork/docs/`
119	+	- Backups: `/opt/makenotwork/backups/`
120	+	- Systemd unit: `/etc/systemd/system/makenotwork.service`
121	+	- Caddy config: `/etc/caddy/Caddyfile`
122	+	- Error pages: `/opt/makenotwork/error-pages/`
123	+	- Git repos: `/opt/git/`
124	+	- Logs: `journalctl -u makenotwork -f`
125	+	- Port: 127.0.0.1:3000 (Caddy reverse proxies from 443)
126	+	- DB: PostgreSQL `makenotwork` database, `makenotwork` user (peer auth)
127	+	- Restart policy: `Restart=always`, `RestartSec=5`

A server/docs/schema.md +500

		@@ -0,0 +1,512 @@
1	+	# Schema — MNW Server
2	+
3	+	PostgreSQL database. 57 migrations in `migrations/`, auto-applied on boot via sqlx. Extension: `pg_trgm` (trigram fuzzy search).
4	+
5	+	## Domain Map
6	+
7	+	\| Domain \| Tables \| Purpose \|
8	+	\|--------\|--------\|---------\|
9	+	\| Users & Auth \| 6 \| Accounts, passkeys, sessions, 2FA, login tokens \|
10	+	\| Projects & Content \| 8 \| Creator projects, items, versions, chapters, insertions, sections, bundles \|
11	+	\| Tags & Taxonomy \| 4 \| Hierarchical tags, item tagging, platform labels \|
12	+	\| Commerce \| 7 \| Transactions, subscriptions, promo codes, license keys \|
13	+	\| Creator Tiers \| 2 \| Platform subscription tiers (Basic/Small/Big/Streaming) \|
14	+	\| Email & Mailing \| 4 \| Mailing lists, subscribers, suppressions, signups \|
15	+	\| Social \| 3 \| Follows, blog posts, custom links \|
16	+	\| Collections \| 2 \| User-curated item lists \|
17	+	\| SyncKit \| 5 \| Cloud sync apps, devices, keys, changelog, blobs \|
18	+	\| Git \| 6 \| Repos, SSH keys, issues, comments, labels \|
19	+	\| OTA Updates \| 4 \| Releases, artifacts, build configs, build runs \|
20	+	\| Custom Domains \| 1 \| Creator vanity domains \|
21	+	\| OAuth \| 1 \| PKCE authorization codes \|
22	+	\| Waitlist & Invites \| 3 \| Creator waves, waitlist, invite codes \|
23	+	\| Content Security \| 2 \| Download fingerprints, streaming sessions \|
24	+	\| Admin \| 1 \| Abuse reports \|
25	+	\| Media \| 1 \| User media library (images for markdown) \|
26	+	\| Import \| 1 \| Bulk import jobs (Patreon, Ko-fi, Gumroad) \|
27	+	\| Sessions \| 1 \| HTTP sessions (tower-sessions) \|
28	+
29	+	---
30	+
31	+	## Users & Authentication
32	+
33	+	### users
34	+	Core accounts. Every user has one row; creator features are gated by `can_create_projects`.
35	+
36	+	\| Column \| Type \| Notes \|
37	+	\|--------\|------\|-------\|
38	+	\| id \| UUID PK \| \|
39	+	\| username \| TEXT UNIQUE \| URL slug (`/@username`) \|
40	+	\| email \| TEXT UNIQUE \| \|
41	+	\| display_name \| TEXT \| \|
42	+	\| password_hash \| TEXT \| argon2 \|
43	+	\| email_verified \| BOOL \| \|
44	+	\| totp_secret / totp_enabled \| TEXT / BOOL \| 2FA \|
45	+	\| failed_login_attempts \| INT \| Resets on success \|
46	+	\| locked_until \| TIMESTAMPTZ \| Lockout after 5 failures \|
47	+	\| stripe_account_id \| TEXT \| Stripe Connect account \|
48	+	\| stripe_onboarding_complete \| BOOL \| \|
49	+	\| can_create_projects \| BOOL \| Creator gate \|
50	+	\| creator_tier \| TEXT \| 'basic', 'small_files', 'big_files', 'streaming' \|
51	+	\| storage_used_bytes \| BIGINT \| Computed from versions + insertions \|
52	+	\| max_file_override_bytes \| BIGINT \| Per-user override \|
53	+	\| grandfathered_until \| TIMESTAMPTZ \| Grace period for existing creators \|
54	+	\| suspended_at \| TIMESTAMPTZ \| Null = active \|
55	+	\| notify_sale / notify_follower / notify_release / notify_issues \| BOOL \| Email prefs \|
56	+
57	+	Indexes: email, username, email_verified, stripe_account — all B-tree.
58	+	Trigger: `update_users_updated_at` — auto-sets `updated_at`.
59	+
60	+	### user_passkeys
61	+	WebAuthn credentials for passwordless login.
62	+
63	+	\| Column \| Type \| Notes \|
64	+	\|--------\|------\|-------\|
65	+	\| id \| UUID PK \| \|
66	+	\| user_id \| UUID FK → users CASCADE \| \|
67	+	\| credential_json \| JSONB \| WebAuthn credential blob \|
68	+	\| credential_id \| BYTEA UNIQUE \| Lookup key \|
69	+	\| name \| TEXT \| User-assigned label \|
70	+
71	+	### login_tokens
72	+	Single-use email login links.
73	+
74	+	- FK: user_id → users CASCADE
75	+	- Key columns: token_hash, expires_at, used_at
76	+	- Indexed on: user_id, expires_at
77	+
78	+	### user_sessions
79	+	Active login sessions. Tracks last activity, UA, IP for "active sessions" UI.
80	+
81	+	- FK: user_id → users CASCADE
82	+	- Indexed on: user_id
83	+
84	+	### backup_codes
85	+	2FA recovery codes (hashed). Marked with `used_at` when consumed.
86	+
87	+	- FK: user_id → users CASCADE
88	+
89	+	### tower_sessions.session
90	+	HTTP session storage (tower-sessions-sqlx-store). Schema `tower_sessions`, PK is TEXT `id`, stores BYTEA `data` with `expiry_date`.
91	+
92	+	---
93	+
94	+	## Projects & Content
95	+
96	+	### projects
97	+	Creator projects — music releases, software, podcasts, books, etc.
98	+
99	+	\| Column \| Type \| Notes \|
100	+	\|--------\|------\|-------\|
101	+	\| id \| UUID PK \| \|
102	+	\| user_id \| UUID FK → users CASCADE \| \|
103	+	\| slug \| TEXT \| URL path segment \|
104	+	\| title \| TEXT \| \|
105	+	\| project_type \| TEXT \| 'music', 'software', 'podcast', etc. \|
106	+	\| is_public \| BOOL \| \|
107	+	\| category_id \| UUID FK → project_categories SET NULL \| \|
108	+	\| mt_community_id \| UUID \| Links to Multithreaded forum \|
109	+	\| features \| TEXT[] \| Feature flags per project \|
110	+
111	+	Constraint: UNIQUE(user_id, slug).
112	+	Indexes: user_id, is_public, category_id, title trigram (GIN), description trigram (GIN).
113	+	Trigger: `update_projects_updated_at`.
114	+
115	+	### items
116	+	Products/content within projects. The central commerce entity — holds pricing, audio, text, licensing.
117	+
118	+	\| Column \| Type \| Notes \|
119	+	\|--------\|------\|-------\|
120	+	\| id \| UUID PK \| \|
121	+	\| project_id \| UUID FK → projects CASCADE \| \|
122	+	\| title / slug \| TEXT \| UNIQUE(project_id, slug) \|
123	+	\| item_type \| TEXT \| 'article', 'audio', 'download', 'video', etc. \|
124	+	\| price_cents \| INT \| 0 = free. CHECK >= 0 \|
125	+	\| pwyw_enabled \| BOOL \| Pay what you want \|
126	+	\| pwyw_min_cents \| INT \| Floor for PWYW \|
127	+	\| body / word_count / reading_time_minutes \| TEXT / INT / INT \| Text content \|
128	+	\| audio_url / audio_s3_key / duration_seconds \| TEXT / TEXT / FLOAT \| Audio content \|
129	+	\| video_s3_key / video_duration_seconds \| TEXT / FLOAT \| Video content \|
130	+	\| enable_license_keys \| BOOL \| DRM gate \|
131	+	\| custom_license_text \| TEXT \| License shown on download \|
132	+	\| sales_count / play_count / download_count \| INT \| Denormalized counters \|
133	+	\| web_only \| BOOL \| Prevents download (streaming only) \|
134	+
135	+	Indexes: project_id, is_public, sales_count, tsvector search (title+description+body), title trigram, desc trigram, (project_id, slug).
136	+	Trigger: `update_items_updated_at`.
137	+
138	+	### versions
139	+	Downloadable file versions per item (software releases, audio stems).
140	+
141	+	- FK: item_id → items CASCADE
142	+	- Key columns: version_number, file_url, s3_key, file_size_bytes, is_current, download_count
143	+	- Constraint: UNIQUE WHERE is_current = true (only one current version per item)
144	+
145	+	### chapters
146	+	Audio/podcast chapter markers. Sorted by `start_seconds`.
147	+
148	+	- FK: item_id → items CASCADE
149	+
150	+	### content_insertions
151	+	Reusable audio clips (ads, intros, outros) uploaded by creators.
152	+
153	+	- FK: user_id → users CASCADE
154	+	- Key columns: title, media_type, storage_key, duration_ms, file_size
155	+
156	+	### content_insertion_placements
157	+	Where insertions attach to items. Position is 'pre_roll', 'mid_roll', or 'post_roll'.
158	+
159	+	- FK: item_id → items CASCADE, insertion_id → content_insertions CASCADE
160	+	- Constraint: UNIQUE(item_id, insertion_id, position, offset_ms)
161	+
162	+	### item_sections
163	+	Tabbed content blocks within items (e.g., "Ingredients", "Instructions", "Changelog").
164	+
165	+	- FK: item_id → items CASCADE
166	+	- Constraint: UNIQUE(item_id, slug)
167	+
168	+	### bundle_items
169	+	Associates items into bundle-type items. CHECK(bundle_id != item_id) prevents self-reference.
170	+
171	+	- PK: (bundle_id, item_id)
172	+	- FK: both → items CASCADE
173	+
174	+	---
175	+
176	+	## Tags & Taxonomy
177	+
178	+	### tags
179	+	Hierarchical tag system. `path` uses dot-notation for materialized paths (e.g., `music.electronic.ambient`).
180	+
181	+	\| Column \| Type \| Notes \|
182	+	\|--------\|------\|-------\|
183	+	\| id \| UUID PK \| \|
184	+	\| parent_id \| UUID FK → tags CASCADE \| Self-referential hierarchy \|
185	+	\| name \| TEXT \| Display name \|
186	+	\| slug \| TEXT UNIQUE \| URL-safe \|
187	+	\| path \| TEXT \| Materialized path (dot-notation) \|
188	+
189	+	Indexes: parent_id, slug, name trigram (GIN), path (prefix queries).
190	+
191	+	### item_tags
192	+	Many-to-many. `is_primary` marks the main tag for an item (UNIQUE WHERE is_primary).
193	+
194	+	- PK: (item_id, tag_id), both CASCADE
195	+
196	+	### labels
197	+	Platform-curated promises (e.g., "DRM-free", "Lossless audio"). Includes definition, examples, and non-examples.
198	+
199	+	- Key columns: slug UNIQUE, display_name, definition, examples, nonexamples
200	+
201	+	### project_labels
202	+	Projects adopt platform labels (creator commits to the promise).
203	+
204	+	- PK: (project_id, label_id), both CASCADE
205	+
206	+	---
207	+
208	+	## Commerce & Payments
209	+
210	+	### transactions
211	+	One-off purchases. Status lifecycle: pending → completed / failed / refunded.
212	+
213	+	\| Column \| Type \| Notes \|
214	+	\|--------\|------\|-------\|
215	+	\| id \| UUID PK \| \|
216	+	\| buyer_id \| UUID FK → users CASCADE \| \|
217	+	\| seller_id \| UUID FK → users SET NULL \| Preserved if seller deletes account \|
218	+	\| item_id \| UUID FK → items SET NULL \| Preserved if item deletes \|
219	+	\| amount_cents \| INT \| CHECK >= 0 \|
220	+	\| platform_fee_cents \| INT \| Always 0 (0% fee model) \|
221	+	\| stripe_checkout_session_id \| TEXT \| Links to Stripe \|
222	+	\| status \| TEXT \| 'pending', 'completed', 'failed', 'refunded' \|
223	+	\| item_title / seller_username \| TEXT \| Denormalized for receipt display \|
224	+
225	+	Constraint: UNIQUE(buyer_id, item_id) WHERE status = 'completed' — prevents double purchase.
226	+	Indexes: buyer_id, seller_id, item_id, status, stripe_session.
227	+
228	+	### subscription_tiers
229	+	Per-project recurring tiers. Each tier has a Stripe product+price.
230	+
231	+	- FK: project_id → projects CASCADE
232	+	- Key columns: name, price_cents, stripe_product_id, stripe_price_id, is_active
233	+
234	+	### subscriptions
235	+	Active subscriber records. UNIQUE(subscriber_id, project_id) WHERE status = 'active'.
236	+
237	+	- FK: subscriber_id → users CASCADE, tier_id → subscription_tiers RESTRICT, project_id → projects CASCADE
238	+	- Note: tier_id uses RESTRICT — cannot delete a tier that has active subscribers
239	+
240	+	### subscription_events
241	+	Webhook events from Stripe. Keyed by `stripe_event_id` (UNIQUE) for idempotency.
242	+
243	+	- FK: subscription_id → subscriptions SET NULL
244	+
245	+	### promo_codes
246	+	Unified discount/free-access/free-trial codes. Purpose-specific CHECK constraints enforce valid field combinations.
247	+
248	+	- FK: creator_id → users CASCADE; item_id, project_id, tier_id all → SET NULL on target delete
249	+	- Key columns: code, code_purpose, discount_type, discount_value, trial_days, max_uses, use_count
250	+	- Constraint: UNIQUE(creator_id, code)
251	+
252	+	### license_keys
253	+	DRM keys for download-limited items. Tracks activation count vs max_activations.
254	+
255	+	- FK: item_id → items CASCADE, owner_id → users CASCADE, transaction_id → transactions SET NULL
256	+	- Key columns: key_code UNIQUE, max_activations, activation_count, revoked_at
257	+
258	+	### license_activations
259	+	Per-machine activations. UNIQUE(license_key_id, machine_id) prevents double-activate on same machine.
260	+
261	+	- FK: license_key_id → license_keys CASCADE
262	+
263	+	---
264	+
265	+	## Creator Tiers
266	+
267	+	### creator_subscriptions
268	+	Platform subscription for creators (Basic $10, Small Files $20, Big Files $30, Streaming $40). One row per creator.
269	+
270	+	- FK: user_id → users CASCADE (UNIQUE)
271	+	- Key columns: tier, status, stripe_subscription_id UNIQUE, grace_enforced_at
272	+
273	+	### fan_plus_subscriptions
274	+	Fan+ consumer subscription (discoverability + collection features). One row per user.
275	+
276	+	- FK: user_id → users CASCADE (UNIQUE)
277	+	- Key columns: status, stripe_subscription_id UNIQUE
278	+
279	+	---
280	+
281	+	## Email & Mailing Lists
282	+
283	+	### mailing_lists
284	+	Per-project lists. Types: 'content' (new releases), 'devlog' (development updates), 'patches' (software patches).
285	+
286	+	- FK: project_id → projects CASCADE
287	+	- Constraint: UNIQUE(project_id, list_type)
288	+
289	+	### mailing_list_subscribers
290	+	Supports both registered users and email-only subscribers. CHECK(user_id NOT NULL OR email NOT NULL).
291	+
292	+	- FK: list_id → mailing_lists CASCADE, user_id → users CASCADE (nullable)
293	+	- Constraints: UNIQUE(list_id, user_id), UNIQUE(list_id, email) WHERE email NOT NULL
294	+
295	+	### email_suppressions
296	+	Hard bounces and spam complaints. Prevents sending to known-bad addresses.
297	+
298	+	- Key columns: email UNIQUE (case-insensitive), reason ('HardBounce', 'SpamComplaint')
299	+
300	+	### email_signups
301	+	Landing page "notify me" signups (pre-launch or feature waitlist).
302	+
303	+	- Key columns: email UNIQUE, source
304	+
305	+	---
306	+
307	+	## Social & Community
308	+
309	+	### follows
310	+	Polymorphic follow system — users can follow users, projects, or tags.
311	+
312	+	- FK: follower_id → users CASCADE
313	+	- Key columns: target_type ('user', 'project', 'tag'), target_id
314	+	- Constraint: UNIQUE(follower_id, target_type, target_id)
315	+
316	+	### blog_posts
317	+	Project-level blog posts (devlogs, announcements). Markdown source + rendered HTML.
318	+
319	+	- FK: project_id → projects CASCADE, author_id → users (no cascade)
320	+	- Key columns: title, slug, body_markdown, body_html, published_at, mt_thread_id
321	+	- Constraint: UNIQUE(project_id, slug)
322	+
323	+	### custom_links
324	+	Creator profile links (social, merch, website). Ordered by sort_order.
325	+
326	+	- FK: user_id → users CASCADE
327	+	- Trigger: `update_custom_links_updated_at`
328	+
329	+	---
330	+
331	+	## Collections
332	+
333	+	### collections
334	+	User-curated lists (playlists, reading lists, favorites).
335	+
336	+	- FK: user_id → users CASCADE
337	+	- Constraint: UNIQUE(user_id, slug)
338	+
339	+	### collection_items
340	+	Items in collections. Ordered by `position`.
341	+
342	+	- PK: (collection_id, item_id), both CASCADE
343	+
344	+	---
345	+
346	+	## SyncKit
347	+
348	+	### sync_apps
349	+	Registered SyncKit applications (GO, BB, AF, third-party).
350	+
351	+	- FK: creator_id → users CASCADE; project_id → projects SET NULL, item_id → items SET NULL
352	+	- Key columns: name, api_key UNIQUE, slug, is_active, redirect_uris TEXT[]
353	+
354	+	### sync_devices
355	+	User devices per app. Last-seen tracking for device management UI.
356	+
357	+	- FK: app_id → sync_apps CASCADE, user_id → users CASCADE
358	+	- Constraint: UNIQUE(app_id, user_id, device_name)
359	+
360	+	### sync_keys
361	+	E2E encryption keys per app/user pair. Key rotation via `key_version`.
362	+
363	+	- PK: (app_id, user_id), both CASCADE
364	+	- Key columns: key_version, encrypted_key
365	+
366	+	### sync_log
367	+	Changelog of data operations. Sequential `seq` (BIGSERIAL) enables cursor-based pull.
368	+
369	+	- FK: app_id, user_id, device_id — all CASCADE
370	+	- Key columns: table_name, operation ('INSERT'/'UPDATE'/'DELETE'), row_id, data (JSONB), client_timestamp
371	+	- Index: (app_id, user_id, seq) — the primary pull query path
372	+
373	+	### sync_blobs
374	+	File blobs for SyncKit (content-hashed dedup). Used when `sync_files=true` on a VFS.
375	+
376	+	- FK: app_id, user_id — both CASCADE
377	+	- Constraint: UNIQUE(app_id, user_id, hash)
378	+	- Key columns: hash, s3_key, size_bytes
379	+
380	+	---
381	+
382	+	## Git Integration
383	+
384	+	### git_repos
385	+	Bare git repositories. Displayed via the `/source/` browser (G1, git2-based).
386	+
387	+	- FK: user_id → users CASCADE, project_id → projects SET NULL
388	+	- Constraint: UNIQUE(user_id, name)
389	+
390	+	### ssh_keys
391	+	SSH public keys for git push access.
392	+
393	+	- FK: user_id → users CASCADE
394	+	- Constraint: UNIQUE(user_id, fingerprint)
395	+
396	+	### issues
397	+	Lightweight issue tracker per repo. Sequential `number` per repo.
398	+
399	+	- FK: repo_id → git_repos CASCADE, author_user_id → users CASCADE
400	+	- Constraint: UNIQUE(repo_id, number)
401	+	- Indexed on: (repo_id, status), author
402	+
403	+	### issue_comments
404	+	- FK: issue_id → issues CASCADE, author_user_id → users CASCADE
405	+
406	+	### issue_labels
407	+	Per-repo label definitions with color.
408	+
409	+	- FK: repo_id → git_repos CASCADE
410	+	- Constraint: UNIQUE(repo_id, name)
411	+
412	+	### issue_label_assignments
413	+	- PK: (issue_id, label_id), both CASCADE
414	+
415	+	---
416	+
417	+	## OTA Updates
418	+
419	+	### ota_releases
420	+	App versions published for over-the-air updates (Tauri-compatible protocol).
421	+
422	+	- FK: app_id → sync_apps CASCADE
423	+	- Constraint: UNIQUE(app_id, version)
424	+
425	+	### ota_artifacts
426	+	Binary artifacts per release. One per target/arch combination.
427	+
428	+	- FK: release_id → ota_releases CASCADE
429	+	- Constraint: UNIQUE(release_id, target, arch)
430	+	- Key columns: s3_key, file_size
431	+
432	+	### ota_build_configs
433	+	Automated build configuration per app.
434	+
435	+	- FK: app_id → sync_apps CASCADE (UNIQUE), repo_id → git_repos CASCADE
436	+	- Key columns: build_command, artifact_path, signing_key_path, targets TEXT[], enabled
437	+
438	+	### ota_builds
439	+	Individual build runs. Status lifecycle: pending → building → succeeded / failed.
440	+
441	+	- FK: config_id → ota_build_configs CASCADE, release_id → ota_releases SET NULL
442	+
443	+	---
444	+
445	+	## Remaining Tables
446	+
447	+	### custom_domains
448	+	Creator vanity domains. Verified via DNS TXT record.
449	+
450	+	- FK: user_id → users CASCADE
451	+	- Key columns: domain UNIQUE, verified, verification_token
452	+
453	+	### oauth_authorization_codes
454	+	OAuth PKCE codes for SyncKit SDK clients. Short-lived (5 min), single-use.
455	+
456	+	- FK: app_id → sync_apps CASCADE, user_id → users CASCADE
457	+	- Key columns: code UNIQUE, code_challenge, redirect_uri, expires_at, used_at
458	+
459	+	### creator_waves / creator_waitlist / invite_codes
460	+	Creator onboarding pipeline: waves (batches), waitlist (applications), invite codes (referrals).
461	+
462	+	- creator_waves: wave_number UNIQUE
463	+	- creator_waitlist: user_id → users CASCADE (UNIQUE), wave_id → creator_waves SET NULL
464	+	- invite_codes: creator_id → users CASCADE, code UNIQUE
465	+
466	+	### download_fingerprints
467	+	Watermark tracking for paid downloads. Records fingerprint ID, IP, UA per download.
468	+
469	+	### streaming_sessions
470	+	Active streaming sessions with IP binding and concurrency tracking.
471	+
472	+	### reports
473	+	User abuse reports. Status: open → resolved/dismissed.
474	+
475	+	### media_files
476	+	User media library for embedding in markdown content. S3-backed.
477	+
478	+	- FK: user_id → users CASCADE
479	+	- Constraint: UNIQUE(user_id, folder, filename)
480	+
481	+	### import_jobs
482	+	Bulk import from external platforms (Patreon, Ko-fi, Gumroad). Tracks progress rows.
483	+
484	+	### project_categories
485	+	Taxonomy for project categorization. Referenced by projects.category_id.
486	+
487	+	---
488	+
489	+	## Cascade Summary
490	+
491	+	CASCADE (delete parent → delete children): Most FK relationships. Deleting a user cascades to all their projects, items, content, sync data, sessions, keys, etc.
492	+
493	+	SET NULL (delete parent → null the FK): Used where the child record should survive: transactions keep seller/item info (denormalized title/username), sync_apps keep project/item links optional, git repos keep project association optional.
494	+
495	+	RESTRICT (prevent parent delete): subscription_tiers — cannot delete a tier that has active subscribers.
496	+
497	+	## Search Infrastructure
498	+
499	+	\| Target \| Index Type \| Columns \|
500	+	\|--------\|-----------\|---------\|

Lines truncated

A server/docs/smoke_test.md +89

		@@ -0,0 +1,89 @@
1	+	# Smoke Test Checklist — MNW Server
2	+
3	+	Pre-release manual verification. Run after every production deploy.
4	+
5	+	## Prerequisites
6	+
7	+	- Fresh deploy completed (`deploy.sh` or `deploy.sh --quick`)
8	+	- `systemctl status makenotwork` shows active
9	+	- `curl -s http://127.0.0.1:3000` returns HTML
10	+
11	+	## Core Flows
12	+
13	+	### Auth
14	+	- [ ] Sign up with new account (valid email, password meets requirements)
15	+	- [ ] Log out
16	+	- [ ] Log in with the new account
17	+	- [ ] Password reset: request reset, verify email arrives (or check Postmark Activity)
18	+	- [ ] Rate limiting: 6+ rapid login attempts returns 429
19	+
20	+	### Creator Setup
21	+	- [ ] Enable creator mode on account
22	+	- [ ] Set creator profile (name, bio, avatar)
23	+	- [ ] Stripe Connect onboarding link works (or verify Stripe Dashboard shows pending account)
24	+
25	+	### Content
26	+	- [ ] Create a new project
27	+	- [ ] Add a text item to the project
28	+	- [ ] Upload an audio file (< 500 MB limit)
29	+	- [ ] Upload an image (< 10 MB limit)
30	+	- [ ] Verify uploaded files are accessible via download URL
31	+	- [ ] Edit item metadata (title, description, tags)
32	+	- [ ] Delete an item
33	+
34	+	### Purchase Flow
35	+	- [ ] Create a free item, verify it's downloadable without payment
36	+	- [ ] Create a paid item, verify checkout redirects to Stripe
37	+	- [ ] Complete test checkout (use Stripe test card `4242 4242 4242 4242`)
38	+	- [ ] Verify purchase receipt email arrives (or check Postmark)
39	+	- [ ] Verify buyer can download after purchase
40	+
41	+	### Fan+
42	+	- [ ] Subscribe to a creator's Fan+ tier (test card)
43	+	- [ ] Verify Fan+ badge appears on subscriber profile
44	+	- [ ] Cancel subscription, verify access revokes at period end
45	+
46	+	### Email Verification
47	+	- [ ] Change email address on account
48	+	- [ ] Verification email arrives (check Postmark Activity if needed)
49	+	- [ ] Click verification link, confirm email updates
50	+
51	+	### SyncKit
52	+	- [ ] Authenticate via `/api/sync/auth` with valid credentials
53	+	- [ ] Register a device via `/api/sync/devices`
54	+	- [ ] Push a small changelog batch
55	+	- [ ] Pull changes back, verify data matches
56	+	- [ ] Verify JWT expiration (7-day max)
57	+
58	+	### Git Browser
59	+	- [ ] Navigate to `/source/` — repository list loads
60	+	- [ ] Click a repo, browse files
61	+	- [ ] View a file's content
62	+	- [ ] View commit log
63	+
64	+	### Public Pages
65	+	- [ ] Homepage loads (`/`)
66	+	- [ ] Creator profile page loads (`/@username`)
67	+	- [ ] Project page loads
68	+	- [ ] Documentation pages load (`/docs/...`)
69	+	- [ ] RSS feed loads (`/@username/rss`)
70	+
71	+	## Infrastructure Checks
72	+
73	+	- [ ] `systemctl status makenotwork` — active, no recent restarts
74	+	- [ ] `systemctl status caddy` — active
75	+	- [ ] `systemctl status postgresql` — active
76	+	- [ ] `journalctl -u makenotwork --since "10 min ago"` — no errors
77	+	- [ ] `curl -s https://makenot.work` — responds with 200 (through Cloudflare)
78	+	- [ ] DB backup exists: `ls -lt /opt/makenotwork/backups/ \| head -3`
79	+
80	+	## Webhook Verification
81	+
82	+	- [ ] Stripe Dashboard → Webhooks → recent deliveries show 200 responses
83	+	- [ ] If testing: `stripe trigger checkout.session.completed` → server processes event
84	+
85	+	## Post-Deploy Spot Checks
86	+
87	+	- [ ] Static assets load (CSS, JS, fonts) — no broken styles
88	+	- [ ] Custom error pages work: visit a non-existent URL, verify 404 page renders
89	+	- [ ] File upload S3 connectivity: upload a small test file

A server/docs/test_plan.md +144

		@@ -0,0 +1,144 @@
1	+	# Test Plan — MNW Server
2	+
3	+	## Overview
4	+
5	+	~3,200 tests total. ~66,000 lines of test code. Unit tests (no DB) + integration tests (real PostgreSQL).
6	+
7	+	## Test Architecture
8	+
9	+	Unit tests: Inline `#[cfg(test)]` modules in source files. No DB needed.
10	+
11	+	Integration tests: `tests/workflows/` directory. Each test gets a fresh PostgreSQL database via the test harness. Tests are grouped by feature domain.
12	+
13	+	Test harness: `tests/harness/` provides `TestHarness` — spins up a real app instance with its own database, HTTP client, and helper methods (signup, login, create project, etc.).
14	+
15	+	DB creation pattern: Each integration test calls `TestHarness::new().await`, which:
16	+	1. Connects to the `postgres` database (from `TEST_DATABASE_URL`)
17	+	2. Creates a unique `mnw_test_{uuid}` database
18	+	3. Runs all sqlx migrations
19	+	4. Boots the full app router
20	+	5. Drops the test database on `TestHarness::drop()`
21	+
22	+	CI: `server/deploy/run-ci.sh` runs on astra. Executes `cargo check`, `cargo test`, `cargo clippy`, `cargo audit`.
23	+
24	+	## Running Tests
25	+
26	+	```bash
27	+	# Unit tests only (no DB needed)
28	+	cargo test --lib
29	+
30	+	# Integration tests (needs PostgreSQL)
31	+	TEST_DATABASE_URL="postgres:///postgres" cargo test --test integration
32	+
33	+	# All tests
34	+	TEST_DATABASE_URL="postgres:///postgres" cargo test
35	+
36	+	# Filtered
37	+	TEST_DATABASE_URL="postgres:///postgres" cargo test --test integration auth::
38	+
39	+	# On astra (recommended for integration tests)
40	+	ssh 100.106.221.39
41	+	/home/max/staging/run-tests.sh # All
42	+	/home/max/staging/run-tests.sh auth:: # Filtered
43	+	```
44	+
45	+	Important: Use `--test-threads=8` (or set `RUST_TEST_THREADS=8`). Astra has 96 cores — default parallelism overwhelms PostgreSQL with too many simultaneous `CREATE DATABASE` calls.
46	+
47	+	Orphaned test DBs: If tests are killed mid-run, `mnw_test_*` databases remain. The `run-tests.sh` script cleans these up automatically. Manual cleanup:
48	+	```bash
49	+	psql -t -c "SELECT datname FROM pg_database WHERE datname LIKE 'mnw_test_%';" postgres \
50	+	\| xargs -I{} psql -c "DROP DATABASE IF EXISTS \"{}\";" postgres
51	+	```
52	+
53	+	## What's Covered
54	+
55	+	### Integration Tests (`tests/workflows/`)
56	+
57	+	\| Domain \| Test File(s) \| What's Tested \|
58	+	\|--------\|-------------\|---------------\|
59	+	\| Auth \| `auth.rs`, `adversarial_auth.rs` \| Signup, login, logout, session management, password validation, rate limiting, brute force protection \|
60	+	\| Account \| `account.rs`, `account_deletion.rs` \| Profile update, email change, password change, full account deletion cascade \|
61	+	\| Creator \| `creator.rs`, `creator_media.rs` \| Creator onboarding, profile setup, media uploads \|
62	+	\| Content \| `content.rs`, `content_insertions.rs` \| Item CRUD, content types, insertions, ordering \|
63	+	\| Payments \| `fan_plus.rs` \| Fan+ subscriptions, Stripe checkout flows \|
64	+	\| Collections \| `collections.rs` \| Collection CRUD, item membership \|
65	+	\| Categories \| `categories.rs` \| Category tree management \|
66	+	\| Blog \| `blog.rs` \| Blog post CRUD, rendering \|
67	+	\| Broadcast \| `broadcast.rs` \| Email broadcast campaigns \|
68	+	\| Chapters \| `chapters.rs` \| Chapter management for content \|
69	+	\| Contacts \| `contacts.rs` \| Creator contact list \|
70	+	\| Custom Domains \| `custom_domains.rs` \| Domain verification, routing \|
71	+	\| Custom Links \| `custom_links.rs` \| Creator link management \|
72	+	\| Exports \| `exports.rs` \| Data export functionality \|
73	+	\| Follows \| `follows.rs` \| Creator follow/unfollow \|
74	+	\| Analytics \| `analytics.rs` \| View tracking, download counts \|
75	+	\| Appeal \| `appeal.rs` \| Content moderation appeals \|
76	+	\| Admin \| `admin.rs` \| Admin panel operations \|
77	+	\| Builds \| `builds.rs` \| Software build/release management \|
78	+	\| Git Browser \| `git_browser.rs`, `git_issues.rs`, `git_project.rs` \| Repo browsing, issue tracking, project settings \|
79	+	\| SyncKit \| `synckit.rs` \| Auth, devices, push/pull, key management, validation \|
80	+	\| Fingerprinting \| `fingerprinting.rs` \| Content fingerprinting, license templates \|
81	+	\| Adversarial \| `adversarial.rs`, `adversarial_business.rs`, `adversarial_input.rs` \| XSS, injection, oversized inputs, permission boundaries, business logic abuse \|
82	+
83	+	### Unit Tests (inline)
84	+
85	+	\| Module \| Tests \| What's Tested \|
86	+	\|--------\|-------\|---------------\|
87	+	\| `types/mod.rs` \| 14 \| Type conversions, enum parsing, ID generation \|
88	+	\| `payments/webhooks.rs` \| 22 \| Webhook signature verification, event routing, idempotency \|
89	+	\| `payments/checkout.rs` \| 12 \| Checkout session creation, price calculation, discount logic \|
90	+	\| `fingerprint/license_templates.rs` \| 8 \| License text generation, template variable substitution \|
91	+	\| `csrf.rs` \| — \| CSRF token generation and validation \|
92	+	\| `auth.rs` \| — \| Password hashing, session token generation \|
93	+	\| `config.rs` \| — \| Configuration parsing, env var handling \|
94	+	\| `rss.rs` \| — \| RSS feed generation \|
95	+	\| `scanning/*` \| — \| ClamAV, YARA, hash lookup, content type detection \|
96	+
97	+	## What's Intentionally Not Tested
98	+
99	+	\| Area \| Reason \|
100	+	\|------\|--------\|
101	+	\| Stripe live mode \| Would charge real money. Test mode webhooks are tested. \|
102	+	\| Postmark delivery \| Would send real emails. Email composition logic is tested; delivery is mocked. \|
103	+	\| Cloudflare CDN \| Infrastructure layer, not application logic. \|
104	+	\| TLS/certificate handling \| Caddy handles this; tested manually. \|
105	+	\| Systemd integration \| Deployment concern, not application logic. \|
106	+	\| Real S3 uploads \| Integration tests mock S3 or use test endpoints. \|
107	+	\| ClamAV/YARA scanning \| Requires running daemons; scanning logic is unit-tested. \|
108	+
109	+	## Adding New Tests
110	+
111	+	### Integration test pattern
112	+	```rust
113	+	#[tokio::test]
114	+	async fn my_new_test() {
115	+	let mut h = TestHarness::new().await;
116	+	let user_id = h.signup("testuser", "test@example.com", "Password1!").await;
117	+
118	+	// Use h.client for HTTP requests
119	+	let resp = h.client.get("/api/some-endpoint").await;
120	+	assert_eq!(resp.status, 200);
121	+
122	+	// Or use h.db for direct SQL
123	+	let row = sqlx::query("SELECT ...").fetch_one(&h.db).await.unwrap();
124	+	}
125	+	```
126	+
127	+	### Unit test pattern
128	+	```rust
129	+	#[cfg(test)]
130	+	mod tests {
131	+	use super::*;
132	+
133	+	#[test]
134	+	fn my_unit_test() {
135	+	// No DB, no async, pure logic
136	+	}
137	+	}
138	+	```
139	+
140	+	## Key Paths
141	+
142	+	- `tests/harness/` — TestHarness, TestClient, helper methods
143	+	- `tests/workflows/` — All integration test files
144	+	- `deploy/run-ci.sh` — CI script (runs on astra)

A server/docs/troubleshooting.md +178

		@@ -0,0 +1,178 @@
1	+	# Troubleshooting — MNW Server
2	+
3	+	## Service Won't Start
4	+
5	+	```
6	+	Check logs: journalctl -u makenotwork -n 50 --no-pager
7	+	```
8	+
9	+	\| Symptom \| Cause \| Fix \|
10	+	\|---------\|-------\|-----\|
11	+	\| "DATABASE_URL environment variable is required" \| Missing env var \| Check `/opt/makenotwork/.env` has `DATABASE_URL` \|
12	+	\| "SIGNING_SECRET is required in production" \| HOST=0.0.0.0 or HTTPS HOST_URL without secret \| Set `SIGNING_SECRET` to a random string in `.env` \|
13	+	\| "Invalid HOST address" / "Invalid PORT number" \| Malformed HOST or PORT \| HOST must be valid IP (default 127.0.0.1), PORT must be integer (default 3000) \|
14	+	\| Startup hangs then fails \| PostgreSQL unreachable \| `systemctl status postgresql`, verify `DATABASE_URL` connection string \|
15	+	\| "Failed to run migrations" \| Migration error or DB permissions \| Connect manually: `psql $DATABASE_URL -c "SELECT 1"`, check migration SQL \|
16	+	\| "Failed to migrate session store" \| tower_sessions table issue \| Usually resolves on retry. If persistent, check DB user has CREATE TABLE permission \|
17	+	\| Startup succeeds but features missing \| Optional services not configured \| Stripe, Postmark, S3, Git browser, SyncKit all degrade gracefully if env vars missing \|
18	+
19	+	## 502 Errors
20	+
21	+	Caddy serves `/opt/makenotwork/error-pages/502.html` when the app is unreachable.
22	+
23	+	1. Is the process running?
24	+	```bash
25	+	systemctl status makenotwork --no-pager
26	+	```
27	+	- Not running → `systemctl restart makenotwork`, check logs for crash cause
28	+	- Running but not responding → check port: `curl -s http://127.0.0.1:3000`
29	+
30	+	2. Is Caddy running?
31	+	```bash
32	+	systemctl status caddy --no-pager
33	+	```
34	+	- Not running → `systemctl restart caddy`
35	+
36	+	3. Is PostgreSQL running?
37	+	```bash
38	+	systemctl status postgresql --no-pager
39	+	sudo -u makenotwork psql makenotwork -c "SELECT 1"
40	+	```
41	+	- Not running → `systemctl restart postgresql`, then `systemctl restart makenotwork`
42	+
43	+	4. Port conflict?
44	+	```bash
45	+	lsof -i :3000
46	+	```
47	+	- Another process → kill it or change `PORT` in `.env`
48	+
49	+	## Slow Queries
50	+
51	+	Symptoms: Pages load slowly, "timeout acquiring connection" in logs, high PostgreSQL CPU.
52	+
53	+	Diagnostics:
54	+	```bash
55	+	# Enable slow query logging
56	+	sudo -u postgres psql -c "ALTER SYSTEM SET log_min_duration_statement = 1000;"
57	+	sudo -u postgres psql -c "SELECT pg_reload_conf();"
58	+
59	+	# Check active queries
60	+	sudo -u postgres psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 5;"
61	+	```
62	+
63	+	Known patterns:
64	+	- Discover search with very short terms → triggers trigram scan. The `pg_trgm` extension + GIN index mitigate this.
65	+	- Tag hierarchy queries → EXISTS subqueries on items with many tags.
66	+	- Connection pool exhaustion → default is 25 connections, 3s acquire timeout. If all busy, new requests fail after 3s.
67	+
68	+	## Stripe Webhook Failures
69	+
70	+	Symptoms: Purchases not completing, subscriptions not updating.
71	+
72	+	1. Check Stripe Dashboard → Webhooks for failed deliveries
73	+	2. Check server logs:
74	+	```bash
75	+	journalctl -u makenotwork --since "1 hour ago" \| grep -i stripe
76	+	```
77	+
78	+	\| Log Message \| Cause \| Fix \|
79	+	\|-------------\|-------\|-----\|
80	+	\| "Missing Stripe signature" \| Request missing `Stripe-Signature` header \| Webhook URL misconfigured in Stripe Dashboard \|
81	+	\| "Invalid payload encoding" \| Non-UTF8 body \| Stripe endpoint URL wrong (hitting wrong service) \|
82	+	\| Signature verification error \| `STRIPE_WEBHOOK_SECRET` mismatch \| Copy exact secret from Stripe Dashboard → Webhooks, update `.env`, restart \|
83	+	\| Event type not handled (debug log) \| Unhandled event type \| Expected — only specific events are processed \|
84	+	\| "Stripe not configured" \| Missing `STRIPE_SECRET_KEY` \| Set env var in `.env`, restart \|
85	+
86	+	Test locally:
87	+	```bash
88	+	stripe listen --forward-to localhost:3000/stripe/webhook
89	+	stripe trigger checkout.session.completed
90	+	```
91	+
92	+	## Email Not Sending
93	+
94	+	Symptoms: Password resets, purchase receipts, or verification emails not arriving.
95	+
96	+	1. Is Postmark configured?
97	+	- Check `.env` for `POSTMARK_TOKEN`. If missing, emails log to stdout (dev mode).
98	+
99	+	2. Is the recipient suppressed?
100	+	```sql
101	+	SELECT * FROM email_suppressions WHERE email = 'user@example.com';
102	+	```
103	+	- If found, remove: `DELETE FROM email_suppressions WHERE email = 'user@example.com';`
104	+
105	+	3. Check Postmark Dashboard → Activity for delivery status
106	+
107	+	4. Check server logs:
108	+	```bash
109	+	journalctl -u makenotwork --since "1 hour ago" \| grep -i email
110	+	```
111	+
112	+	\| Log Message \| Cause \| Fix \|
113	+	\|-------------\|-------\|-----\|
114	+	\| "email skipped (suppressed)" \| Recipient on suppression list \| Remove from `email_suppressions` table \|
115	+	\| "Failed to send email" \| Postmark API error (timeout, auth, invalid address) \| Check Postmark Dashboard for details, verify token \|
116	+	\| Emails logged to console \| `POSTMARK_TOKEN` not set \| Set env var, restart \|
117	+
118	+	## Sync Failures (SyncKit)
119	+
120	+	Symptoms: Desktop apps can't push/pull data.
121	+
122	+	1. Is SyncKit configured?
123	+	- Check `.env` for `SYNCKIT_JWT_SECRET`. If missing, endpoints return 503.
124	+
125	+	2. JWT issues:
126	+
127	+	\| Error \| Cause \| Fix \|
128	+	\|-------\|-------\|-----\|
129	+	\| 401 Unauthorized \| Token expired (7-day max) or bad signature \| Client should re-authenticate via `/api/synckit/auth` \|
130	+	\| "Unknown app" \| API key invalid or app inactive \| Check `sync_apps` table: `SELECT * FROM sync_apps WHERE api_key = '...'` \|
131	+	\| "Unknown device" \| Device not registered \| Client should call `POST /api/sync/devices` first \|
132	+
133	+	3. Push failures:
134	+
135	+	\| Error \| Cause \| Fix \|
136	+	\|-------\|-------\|-----\|
137	+	\| "Maximum 500 changes per push" \| Batch too large \| Client should split into ≤500-change batches \|
138	+	\| "Table name validation failed" \| Invalid chars in table name \| Use alphanumeric + underscores only, max 100 chars \|
139	+	\| "DELETE operations should not include data" \| Data payload on DELETE op \| Client bug — set `data: null` for DELETEs \|
140	+
141	+	4. Blob storage:
142	+	- Check `SYNCKIT_S3_*` env vars for the separate SyncKit bucket
143	+	- If S3 unreachable, blob up/download fails but changelog sync still works
144	+
145	+	## Git Browser Errors
146	+
147	+	Symptoms: Source browser pages return 404 or 500.
148	+
149	+	1. Is the git browser configured?
150	+	- Check `.env` for `GIT_REPOS_PATH`. If missing, all git routes return 404.
151	+
152	+	2. Repository not found:
153	+	```bash
154	+	ls /opt/git/ # Check bare repos exist
155	+	```
156	+	- Repos must be bare (`git init --bare`)
157	+	- Path structure: `$GIT_REPOS_PATH/{owner}/{repo}.git/`
158	+
159	+	3. File too large (>1MB): Intentional limit. Large files show truncation message.
160	+
161	+	4. Repo corruption:
162	+	```bash
163	+	cd /opt/git/owner/repo.git && git fsck --full
164	+	```
165	+
166	+	## Resource Limits
167	+
168	+	\| Resource \| Limit \| What Happens \|
169	+	\|----------\|-------\|-------------\|
170	+	\| DB connections \| 25 max \| "timeout acquiring connection" after 3s wait \|
171	+	\| Memory \| 512M (systemd MemoryMax) \| Process killed by OOM, auto-restarts \|
172	+	\| File descriptors \| 65535 (LimitNOFILE) \| "too many open files" \|
173	+	\| File upload: audio \| 500 MB \| 413 Payload Too Large \|
174	+	\| File upload: image \| 10 MB \| 413 Payload Too Large \|
175	+	\| File upload: video \| 20 GB \| 413 Payload Too Large \|
176	+	\| Login rate limit \| 2/sec, burst 5 \| 429 Too Many Requests \|
177	+	\| API rate limit \| 2/sec, burst 10 \| 429 Too Many Requests \|
178	+	\| SyncKit rate limit \| 10/sec, burst 30 \| 429 Too Many Requests \|