| 1 |
# Deploying multithreaded |
| 2 |
|
| 3 |
Two instances, two mechanisms, because they are different architectures. |
| 4 |
|
| 5 |
|
| 6 |
|
| 7 |
| staging demo | testnot-1 | x86_64 | Sando, as a companion of the MNW server | |
| 8 |
| production | alpha-west-1 | x86_64 | Sando, as a companion of the MNW server | |
| 9 |
| astra | astra | aarch64 | `deploy/deploy.sh` | |
| 10 |
|
| 11 |
testnot-1 carries a **read-only demo**, provisioned 2026-08-06. It is seeded from |
| 12 |
mt's own `--seed`, holds no prod data, and is read-only by construction rather |
| 13 |
than by a flag: every write path needs a session, a session needs a completed |
| 14 |
OAuth login, and no OAuth client is registered there. Unauthenticated POSTs come |
| 15 |
back 403. |
| 16 |
|
| 17 |
That is what makes mt browser-testable at all — the read half of it. Before this, |
| 18 |
every audit run skipped its browser axis, so mt template and frontend work shipped |
| 19 |
without the verification every other project gets. Write coverage still needs the |
| 20 |
astra harness instance below. |
| 21 |
|
| 22 |
Sando builds only on its configured `build_host` (fw13, x86_64) and refuses to |
| 23 |
compile anywhere else, so it cannot produce the aarch64 binary astra needs. That |
| 24 |
is the same constraint that sent pom to Bento. Astra keeps its own script. |
| 25 |
|
| 26 |
## Production: a Sando companion |
| 27 |
|
| 28 |
There is no multithreaded-specific deploy command. mt is built from the same |
| 29 |
worktree and sha as the MNW server, staged into the same release bundle, and |
| 30 |
installed on `testnot-1` and `prod-1` after the server's symlink swap. Run a |
| 31 |
normal MNW deploy and mt goes with it. |
| 32 |
|
| 33 |
What that buys, none of which the old script had: a native build on fw13, the |
| 34 |
`migration_dry_run` gate against mt's own prod dump, the `cargo_test` gate, the |
| 35 |
frontend build treated as fatal rather than as a `cargo::warning`, a 48h burn-in |
| 36 |
on testnot, `node_health`, and a binary that ships from the same sha as the |
| 37 |
server it talks to. |
| 38 |
|
| 39 |
Config lives in two files in `sando/`, split the way Sando splits everything: |
| 40 |
`daemon/sando-daemon.toml` (and `deploy/sando-daemon.toml.example`) says mt is a |
| 41 |
`[[companion]]` to build, and `sando.toml` says which nodes install it. |
| 42 |
|
| 43 |
The cost is that `Cargo.toml`'s version goes decorative, as mnw-cli's already |
| 44 |
has. mt is not distributed on its own, so an independent version buys nothing. |
| 45 |
Releases are identified by the bundle digest and the sha. |
| 46 |
|
| 47 |
### Why the binary has no `static/` beside it |
| 48 |
|
| 49 |
mt embeds its `static/` tree (`src/static_assets.rs`) instead of serving it from |
| 50 |
a directory next to the binary. A companion is exactly one installed file, and |
| 51 |
`build.rs` bakes a content hash of those assets into the templates compiled into |
| 52 |
the binary — so a binary-only install would serve new markup against old CSS |
| 53 |
with nothing reporting an error. Embedding makes the two inseparable. |
| 54 |
|
| 55 |
Consequence for anyone editing assets: a change to `static/style.css`, `mt.js` |
| 56 |
or the TypeScript under `frontend/src` needs a rebuild to take effect. It |
| 57 |
already did, since those are what the `?v=` cache-buster hashes. |
| 58 |
|
| 59 |
### One-time node setup |
| 60 |
|
| 61 |
None outstanding. prod-1 has carried `/usr/local/lib/mnw/install-companion.sh` |
| 62 |
and the `/etc/sudoers.d/mnw-companion` grant since mnw-cli, and the grant is on |
| 63 |
the script rather than on a particular companion. testnot-1 got both on |
| 64 |
2026-08-06, along with the postgres role and database, the system user, |
| 65 |
`/opt/multithreaded/.env`, and the unit. |
| 66 |
|
| 67 |
One testnot-only wrinkle worth recording, because it cost a debugging round and |
| 68 |
will bite the next node set up the same way. testnot's `pg_hba.conf` has |
| 69 |
`local all all peer map=mnw`, where prod has a plain `local all all peer`. A map |
| 70 |
does **not** imply the identity pairing, so an OS user connecting as its own |
| 71 |
same-named role is refused (`28000 Peer authentication failed`) until it is |
| 72 |
listed in `pg_ident.conf`. `postgres:///multithreaded` therefore needed: |
| 73 |
|
| 74 |
``` |
| 75 |
mnw multithreaded multithreaded |
| 76 |
``` |
| 77 |
|
| 78 |
Any future node hosting mt needs both files, as root, from `sando/deploy/`: |
| 79 |
|
| 80 |
``` |
| 81 |
sudo install -d /usr/local/lib/mnw |
| 82 |
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh |
| 83 |
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion |
| 84 |
sudo visudo -cf /etc/sudoers.d/mnw-companion |
| 85 |
``` |
| 86 |
|
| 87 |
The sudoers file names every deploy user Sando SSHes as — `makenotwork` for |
| 88 |
prod-1, `deploy` for testnot-1 — so it installs verbatim and needs no hand-edit. |
| 89 |
A node Sando reaches as some other user needs that user added to the file, or the |
| 90 |
install is denied, and denied late: companions install after the symlink swap. |
| 91 |
Check the grant by hand before relying on it: |
| 92 |
|
| 93 |
``` |
| 94 |
ssh <deploy-user>@<node> sudo /usr/local/lib/mnw/install-companion.sh |
| 95 |
``` |
| 96 |
|
| 97 |
Exit 2 with the usage line means the grant works. A password prompt or a "not |
| 98 |
allowed" means it does not. |
| 99 |
|
| 100 |
mt also needs what it has always needed on the box, none of which Sando touches: |
| 101 |
the `multithreaded` postgres role and database, `/opt/multithreaded/.env`, |
| 102 |
`multithreaded.service` installed and enabled (`deploy/multithreaded.service`), |
| 103 |
and a working OS trust store (see below). |
| 104 |
Sando installs the binary and restarts the unit; it does not create users, |
| 105 |
databases, env files, or units, and it will not rewrite a config that drifted. |
| 106 |
|
| 107 |
### The OS trust store is a precondition |
| 108 |
|
| 109 |
mt ships no trust anchors of its own. Every outbound TLS connection it makes |
| 110 |
reads the host CA bundle, which on a Debian-family box means `/etc/ssl/certs` |
| 111 |
kept current by the `ca-certificates` package. Three paths depend on it: the |
| 112 |
OAuth token exchange with the MNW server (`src/auth.rs`), link previews |
| 113 |
(`src/link_preview.rs`), and S3 or MinIO media through `s3-storage`. `reqwest` |
| 114 |
reads the store through `rustls-platform-verifier` and the AWS client through |
| 115 |
`rustls-native-certs`; neither offers bundled roots as an option any more, so |
| 116 |
there is no in-binary fallback to fall back to. |
| 117 |
|
| 118 |
A host with a thin or stale bundle does not fail in a way that looks like a |
| 119 |
deploy problem. mt starts, `/api/health` passes because it only probes the |
| 120 |
database, and pages serve. The first outbound TLS request is a login attempt, |
| 121 |
because mt is an OAuth relying party, so the symptom is a login outage with a |
| 122 |
certificate error buried in the journal on a node that reports healthy. |
| 123 |
|
| 124 |
Two things now make that visible. mt probes the store at startup and logs an |
| 125 |
error when it comes back empty, and `/api/health` carries a `tls_trust_anchors` |
| 126 |
field that PoM asserts, so the condition surfaces as a degraded target rather |
| 127 |
than as a support ticket. Neither watches the host bundle for staleness over |
| 128 |
time; that is infra's job, and keeping `ca-certificates` on an update path is |
| 129 |
still a precondition rather than something mt arranges. |
| 130 |
|
| 131 |
Rollback is the MNW server's rollback: the previous bundle still holds the |
| 132 |
previous mt binary. |
| 133 |
|
| 134 |
## astra: the write-enabled harness instance |
| 135 |
|
| 136 |
`deploy/deploy.sh` installs mt on astra. That instance is not a second staging |
| 137 |
box in general; it exists so an audit run's browser lens can **write**. |
| 138 |
|
| 139 |
The public demo (testnot) is deliberately read-only and no-login, which leaves |
| 140 |
every mutation path browser-unverified, and mutation paths are where |
| 141 |
rendered-behaviour bugs live: the CSRF token round-tripping through a real form |
| 142 |
submission, HX swap targets, optimistic UI, the compiled islands once |
| 143 |
`static/dist` is loaded. Verifying those needs a session, and a session needs a |
| 144 |
real OAuth login. So the harness instance is reachable over the tailnet only, |
| 145 |
logs in against testnot's MNW, and is wiped between runs. |
| 146 |
|
| 147 |
Reached at `https://astra.tailc6b3e1.ts.net:3443`, published by `tailscale serve` |
| 148 |
(`tailscale serve --bg --https=3443 http://127.0.0.1:3400`) and reachable from |
| 149 |
the tailnet only. mt itself binds `127.0.0.1`, so the app port is on no other |
| 150 |
interface and the tailnet proxy is the only way in. |
| 151 |
|
| 152 |
TLS is not optional here, and finding that out is worth two minutes of someone |
| 153 |
else's time: `config.rs` refuses to boot a non-loopback deployment on plain |
| 154 |
`http`, because that would carry OAuth codes and session cookies in the clear. |
| 155 |
`tailscale serve` is what supplies the certificate — astra's own Caddy already |
| 156 |
holds `:443` for something else, hence the `:3443`. Set `COOKIE_SECURE=true` |
| 157 |
to match. |
| 158 |
|
| 159 |
### What it depends on, on the MNW side |
| 160 |
|
| 161 |
Login is the whole point, so mt is only half the setup. The other half is |
| 162 |
testnot's example seed, whose harness phase creates the accounts and the OAuth |
| 163 |
client (`server/src/seed/harness.rs`): |
| 164 |
|
| 165 |
|
| 166 |
|
| 167 |
| `OAUTH_CLIENT_ID` | `mt-astra-harness` (a `sync_apps` row; PKCE, no secret) | |
| 168 |
| Registered redirect | `https://astra.tailc6b3e1.ts.net:3443/auth/callback` | |
| 169 |
| Accounts | `harness_fan` (Fan+), `harness_creator` (creator), `harness_owner` | |
| 170 |
| Password | shared, from `MT_HARNESS_PASSWORD` on the testnot box | |
| 171 |
|
| 172 |
That phase runs only when `MT_HARNESS_PASSWORD` and `MT_HARNESS_REDIRECT_URI` |
| 173 |
are both set in testnot's `EnvironmentFile` (`/etc/mnw/makenotwork.env`), which |
| 174 |
`mnw-testnot-seed.sh` already passes through. Neither value is in the repo: one |
| 175 |
is a credential, the other names a tailnet host. |
| 176 |
|
| 177 |
`SYNCKIT_JWT_SECRET` has to be set on that box too, and it was not until |
| 178 |
2026-08-15. It is the HS256 key `/oauth/token` signs with, and without it every |
| 179 |
token exchange answers 503 "SyncKit is not configured" — a login outage that |
| 180 |
looks like a transport failure from mt's side, since mt retries a 5xx twice |
| 181 |
before giving up. The seed does not set it (it is a machine secret, not seed |
| 182 |
data), so a rebuilt testnot needs it added by hand: 32 characters minimum, or |
| 183 |
the server refuses to boot. |
| 184 |
|
| 185 |
The three accounts carry fixed MNW ids, and mt's own seed (`src/seed.rs`) |
| 186 |
assigns their community roles by `mnw_account_id` — Owner and Moderator for |
| 187 |
`harness_owner` — because login upserts identity and perks but never |
| 188 |
membership. The two id lists have to be edited together; the MNW side has a test |
| 189 |
pinning the literals. |
| 190 |
|
| 191 |
### Walking it |
| 192 |
|
| 193 |
``` |
| 194 |
deploy/harness-walk.sh |
| 195 |
``` |
| 196 |
|
| 197 |
Logs in over OAuth as two harness accounts and walks the write paths the |
| 198 |
instance exists to cover: thread create, reply, flag, and one moderation action, |
| 199 |
each read back afterwards rather than trusted from its redirect. Run it after a |
| 200 |
deploy to this box, and after any change to the login path on either side. |
| 201 |
|
| 202 |
It is the check that covers what unit tests structurally cannot. The first run |
| 203 |
of it found that mt had been sending the OAuth token exchange as JSON while the |
| 204 |
server's `/oauth/token` takes `axum::Form`, so every login answered |
| 205 |
`token_exchange_failed` on a 415 — in production too, since the server changed |
| 206 |
extractors in April. mt's own tests passed throughout, because the wiremock |
| 207 |
stand-in matched on path alone and answered any content type. The mocks now pin |
| 208 |
the content type, and `callback_exchanges_the_code_as_a_form_and_logs_in` covers |
| 209 |
the success path that had no test at all. |
| 210 |
|
| 211 |
Credentials come from `~/.config/mnw/mt-harness.env` (mode 600, not in the |
| 212 |
repo). Exit 0 means a browser lens can log in and write here. |
| 213 |
|
| 214 |
### Resetting it |
| 215 |
|
| 216 |
``` |
| 217 |
deploy/reset-astra.sh |
| 218 |
``` |
| 219 |
|
| 220 |
Stops the service, drops every schema, re-migrates, re-seeds, and then asserts |
| 221 |
the harness preconditions (three accounts, an Owner membership, threads |
| 222 |
present). A run that logs in and finds no membership fails at the first write |
| 223 |
with a 403, which reads like a permissions bug rather than an unfinished seed, |
| 224 |
so the reset verifies rather than assuming. |
| 225 |
|
| 226 |
This resets mt only. The identities live on testnot and come back from its own |
| 227 |
reseed; the two are independent on purpose, because mt's state is disposable and |
| 228 |
the accounts behind it are not. |
| 229 |
|