| 1 |
# Deploying multithreaded |
| 2 |
|
| 3 |
Two instances, two mechanisms, because they are different architectures. |
| 4 |
|
| 5 |
|
| 6 |
|
| 7 |
| staging demo | testnot-1 | x86_64 | Sando, as a companion of the MNW server | |
| 8 |
| production | alpha-west-1 | x86_64 | Sando, as a companion of the MNW server | |
| 9 |
| astra | astra | aarch64 | `deploy/deploy.sh` | |
| 10 |
|
| 11 |
testnot-1 carries a **read-only demo**. It is seeded from |
| 12 |
mt's own `--seed`, holds no prod data, and is read-only by construction rather |
| 13 |
than by a flag: every write path needs a session, a session needs a completed |
| 14 |
OAuth login, and no OAuth client is registered there. Unauthenticated POSTs come |
| 15 |
back 403. |
| 16 |
|
| 17 |
That covers the read half of browser testing. Write coverage needs the astra |
| 18 |
harness instance below. |
| 19 |
|
| 20 |
Sando builds only on its configured `build_host` (fw13, x86_64) and refuses to |
| 21 |
compile anywhere else, so it cannot produce the aarch64 binary astra needs. Astra |
| 22 |
keeps its own script. |
| 23 |
|
| 24 |
## Production: a Sando companion |
| 25 |
|
| 26 |
There is no multithreaded-specific deploy command. mt is built from the same |
| 27 |
worktree and sha as the MNW server, staged into the same release bundle, and |
| 28 |
installed on `testnot-1` and `prod-1` after the server's symlink swap. Run a |
| 29 |
normal MNW deploy and mt goes with it. |
| 30 |
|
| 31 |
That path carries a native build on fw13, the `migration_dry_run` gate against |
| 32 |
mt's own prod dump, the `cargo_test` gate, a fatal frontend build, a 48h burn-in |
| 33 |
on testnot, `node_health`, and a binary that ships from the same sha as the |
| 34 |
server it talks to. |
| 35 |
|
| 36 |
Config lives in two files in `sando/`, split the way Sando splits everything: |
| 37 |
`daemon/sando-daemon.toml` (and `deploy/sando-daemon.toml.example`) says mt is a |
| 38 |
`[[companion]]` to build, and `sando.toml` says which nodes install it. |
| 39 |
|
| 40 |
`Cargo.toml`'s version is decorative. mt is not distributed on its own, so an |
| 41 |
independent version buys nothing. Releases are identified by the bundle digest |
| 42 |
and the sha. |
| 43 |
|
| 44 |
### Why the binary has no `static/` beside it |
| 45 |
|
| 46 |
mt embeds its `static/` tree (`src/static_assets.rs`) instead of serving it from |
| 47 |
a directory next to the binary. A companion is exactly one installed file, and |
| 48 |
`build.rs` bakes a content hash of those assets into the templates compiled into |
| 49 |
the binary, so a binary-only install would serve new markup against old CSS with |
| 50 |
nothing reporting an error. Embedding makes the two inseparable. |
| 51 |
|
| 52 |
Consequence for anyone editing assets: a change to `static/style.css`, `mt.js` |
| 53 |
or the TypeScript under `frontend/src` needs a rebuild to take effect. |
| 54 |
|
| 55 |
### One-time node setup |
| 56 |
|
| 57 |
None outstanding. prod-1 and testnot-1 both carry |
| 58 |
`/usr/local/lib/mnw/install-companion.sh` and the `/etc/sudoers.d/mnw-companion` |
| 59 |
grant, and the grant is on the script rather than on a particular companion. |
| 60 |
Each also needs the postgres role and database, the system user, |
| 61 |
`/opt/multithreaded/.env`, and the unit. |
| 62 |
|
| 63 |
One testnot-only wrinkle. testnot's `pg_hba.conf` has |
| 64 |
`local all all peer map=mnw`, where prod has a plain `local all all peer`. A map |
| 65 |
does **not** imply the identity pairing, so an OS user connecting as its own |
| 66 |
same-named role is refused (`28000 Peer authentication failed`) until it is |
| 67 |
listed in `pg_ident.conf`. `postgres:///multithreaded` therefore needs: |
| 68 |
|
| 69 |
``` |
| 70 |
mnw multithreaded multithreaded |
| 71 |
``` |
| 72 |
|
| 73 |
Any future node hosting mt needs both files, as root, from `sando/deploy/`: |
| 74 |
|
| 75 |
``` |
| 76 |
sudo install -d /usr/local/lib/mnw |
| 77 |
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh |
| 78 |
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion |
| 79 |
sudo visudo -cf /etc/sudoers.d/mnw-companion |
| 80 |
``` |
| 81 |
|
| 82 |
The sudoers file names every deploy user Sando SSHes as (`makenotwork` for |
| 83 |
prod-1, `deploy` for testnot-1), so it installs verbatim and needs no hand-edit. |
| 84 |
A node Sando reaches as some other user needs that user added to the file, or the |
| 85 |
install is denied, and denied late: companions install after the symlink swap. |
| 86 |
Check the grant by hand before relying on it: |
| 87 |
|
| 88 |
``` |
| 89 |
ssh <deploy-user>@<node> sudo /usr/local/lib/mnw/install-companion.sh |
| 90 |
``` |
| 91 |
|
| 92 |
Exit 2 with the usage line means the grant works. A password prompt or a "not |
| 93 |
allowed" means it does not. |
| 94 |
|
| 95 |
mt also needs what it has always needed on the box, none of which Sando touches: |
| 96 |
the `multithreaded` postgres role and database, `/opt/multithreaded/.env`, |
| 97 |
`multithreaded.service` installed and enabled (`deploy/multithreaded.service`), |
| 98 |
and a working OS trust store (see below). |
| 99 |
Sando installs the binary and restarts the unit; it does not create users, |
| 100 |
databases, env files, or units, and it will not rewrite a config that drifted. |
| 101 |
|
| 102 |
### The OS trust store is a precondition |
| 103 |
|
| 104 |
mt ships no trust anchors of its own. Every outbound TLS connection it makes |
| 105 |
reads the host CA bundle, which on a Debian-family box means `/etc/ssl/certs` |
| 106 |
kept current by the `ca-certificates` package. Three paths depend on it: the |
| 107 |
OAuth token exchange with the MNW server (`src/auth.rs`), link previews |
| 108 |
(`src/link_preview.rs`), and S3 or MinIO media through `s3-storage`. `reqwest` |
| 109 |
reads the store through `rustls-platform-verifier` and the AWS client through |
| 110 |
`rustls-native-certs`; neither offers bundled roots as an option any more, so |
| 111 |
there is no in-binary fallback to fall back to. |
| 112 |
|
| 113 |
A host with a thin or stale bundle does not fail in a way that looks like a |
| 114 |
deploy problem. mt starts, `/api/health` passes because it only probes the |
| 115 |
database, and pages serve. The first outbound TLS request is a login attempt, |
| 116 |
because mt is an OAuth relying party, so the symptom is a login outage with a |
| 117 |
certificate error buried in the journal on a node that reports healthy. |
| 118 |
|
| 119 |
Two things make that visible. mt probes the store at startup and logs an error |
| 120 |
when it comes back empty, and `/api/health` carries a `tls_trust_anchors` field |
| 121 |
that PoM asserts, so the condition surfaces as a degraded target rather than as |
| 122 |
a support ticket. Neither watches the host bundle for staleness over time; that |
| 123 |
is infra's job, and keeping `ca-certificates` on an update path is a |
| 124 |
precondition rather than something mt arranges. |
| 125 |
|
| 126 |
Rollback is the MNW server's rollback: the previous bundle still holds the |
| 127 |
previous mt binary. |
| 128 |
|
| 129 |
## astra: the write-enabled harness instance |
| 130 |
|
| 131 |
`deploy/deploy.sh` installs mt on astra. That instance is not a second staging |
| 132 |
box in general; it exists so an audit run's browser lens can **write**. |
| 133 |
|
| 134 |
The public demo (testnot) is deliberately read-only and no-login, which leaves |
| 135 |
every mutation path browser-unverified, and mutation paths are where |
| 136 |
rendered-behaviour bugs live: the CSRF token round-tripping through a real form |
| 137 |
submission, HX swap targets, optimistic UI, the compiled islands once |
| 138 |
`static/dist` is loaded. Verifying those needs a session, and a session needs a |
| 139 |
real OAuth login. So the harness instance is reachable over the tailnet only, |
| 140 |
logs in against testnot's MNW, and is wiped between runs. |
| 141 |
|
| 142 |
Reached at `https://astra.tailc6b3e1.ts.net:3443`, published by `tailscale serve` |
| 143 |
(`tailscale serve --bg --https=3443 http://127.0.0.1:3400`) and reachable from |
| 144 |
the tailnet only. mt itself binds `127.0.0.1`, so the app port is on no other |
| 145 |
interface and the tailnet proxy is the only way in. |
| 146 |
|
| 147 |
TLS is not optional here: `config.rs` refuses to boot a non-loopback deployment |
| 148 |
on plain `http`, because that would carry OAuth codes and session cookies in the |
| 149 |
clear. `tailscale serve` supplies the certificate. astra's own Caddy holds |
| 150 |
`:443` for something else, hence the `:3443`. Set `COOKIE_SECURE=true` to |
| 151 |
match. |
| 152 |
|
| 153 |
### What it depends on, on the MNW side |
| 154 |
|
| 155 |
Login is the whole point, so mt is only half the setup. The other half is |
| 156 |
testnot's example seed, whose harness phase creates the accounts and the OAuth |
| 157 |
client (`server/src/seed/harness.rs`): |
| 158 |
|
| 159 |
|
| 160 |
|
| 161 |
| `OAUTH_CLIENT_ID` | `mt-astra-harness` (a `sync_apps` row; PKCE, no secret) | |
| 162 |
| Registered redirect | `https://astra.tailc6b3e1.ts.net:3443/auth/callback` | |
| 163 |
| Accounts | `harness_fan` (Fan+), `harness_creator` (creator), `harness_owner` | |
| 164 |
| Password | shared, from `MT_HARNESS_PASSWORD` on the testnot box | |
| 165 |
|
| 166 |
That phase runs only when `MT_HARNESS_PASSWORD` and `MT_HARNESS_REDIRECT_URI` |
| 167 |
are both set in testnot's `EnvironmentFile` (`/etc/mnw/makenotwork.env`), which |
| 168 |
`mnw-testnot-seed.sh` already passes through. Neither value is in the repo: one |
| 169 |
is a credential, the other names a tailnet host. |
| 170 |
|
| 171 |
`SYNCKIT_JWT_SECRET` has to be set on that box too. It is the HS256 key |
| 172 |
`/oauth/token` signs with, and without it every token exchange answers 503 |
| 173 |
"SyncKit is not configured", a login outage that looks like a transport failure |
| 174 |
from mt's side, since mt retries a 5xx twice before giving up. The seed does not |
| 175 |
set it (it is a machine secret, not seed data), so a rebuilt testnot needs it |
| 176 |
added by hand: 32 characters minimum, or the server refuses to boot. |
| 177 |
|
| 178 |
The three accounts carry fixed MNW ids, and mt's own seed (`src/seed.rs`) |
| 179 |
assigns their community roles by `mnw_account_id` (Owner and Moderator for |
| 180 |
`harness_owner`), because login upserts identity and perks but never membership. |
| 181 |
The two id lists have to be edited together; the MNW side has a test pinning the |
| 182 |
literals. |
| 183 |
|
| 184 |
### Walking it |
| 185 |
|
| 186 |
``` |
| 187 |
deploy/harness-walk.sh |
| 188 |
``` |
| 189 |
|
| 190 |
Logs in over OAuth as two harness accounts and walks the write paths the |
| 191 |
instance exists to cover: thread create, reply, flag, and one moderation action, |
| 192 |
each read back afterwards rather than trusted from its redirect. Run it after a |
| 193 |
deploy to this box, and after any change to the login path on either side. |
| 194 |
|
| 195 |
It covers what unit tests structurally cannot. mt sends the OAuth token |
| 196 |
exchange as a form, since the server's `/oauth/token` takes `axum::Form` and |
| 197 |
answers 415 to anything else. The wiremock stand-ins pin that content type: a |
| 198 |
mock matching on path alone answers any content type and hides the mismatch. |
| 199 |
|
| 200 |
Credentials come from `~/.config/mnw/mt-harness.env` (mode 600, not in the |
| 201 |
repo). Exit 0 means a browser lens can log in and write here. |
| 202 |
|
| 203 |
### Resetting it |
| 204 |
|
| 205 |
``` |
| 206 |
deploy/reset-astra.sh |
| 207 |
``` |
| 208 |
|
| 209 |
Stops the service, drops every schema, re-migrates, re-seeds, and then asserts |
| 210 |
the harness preconditions (three accounts, an Owner membership, threads |
| 211 |
present). A run that logs in and finds no membership fails at the first write |
| 212 |
with a 403, which reads like a permissions bug rather than an unfinished seed, |
| 213 |
so the reset verifies rather than assuming. |
| 214 |
|
| 215 |
This resets mt only. The identities live on testnot and come back from its own |
| 216 |
reseed; the two are independent on purpose, because mt's state is disposable and |
| 217 |
the accounts behind it are not. |
| 218 |
|