# Deploying multithreaded

Two instances, two mechanisms, because they are different architectures.

| Instance | Host | Arch | Mechanism |
|----------|------|------|-----------|
| staging demo | testnot-1 | x86_64 | Sando, as a companion of the MNW server |
| production | alpha-west-1 | x86_64 | Sando, as a companion of the MNW server |
| astra | astra | aarch64 | `deploy/deploy.sh` |

testnot-1 carries a **read-only demo**, provisioned 2026-08-06. It is seeded from
mt's own `--seed`, holds no prod data, and is read-only by construction rather
than by a flag: every write path needs a session, a session needs a completed
OAuth login, and no OAuth client is registered there. Unauthenticated POSTs come
back 403.

That is what makes mt browser-testable at all — the read half of it. Before this,
every audit run skipped its browser axis, so mt template and frontend work shipped
without the verification every other project gets. Write coverage still needs the
astra harness instance below.

Sando builds only on its configured `build_host` (fw13, x86_64) and refuses to
compile anywhere else, so it cannot produce the aarch64 binary astra needs. That
is the same constraint that sent pom to Bento. Astra keeps its own script.

## Production: a Sando companion

There is no multithreaded-specific deploy command. mt is built from the same
worktree and sha as the MNW server, staged into the same release bundle, and
installed on `testnot-1` and `prod-1` after the server's symlink swap. Run a
normal MNW deploy and mt goes with it.

What that buys, none of which the old script had: a native build on fw13, the
`migration_dry_run` gate against mt's own prod dump, the `cargo_test` gate, the
frontend build treated as fatal rather than as a `cargo::warning`, a 48h burn-in
on testnot, `node_health`, and a binary that ships from the same sha as the
server it talks to.

Config lives in two files in `sando/`, split the way Sando splits everything:
`daemon/sando-daemon.toml` (and `deploy/sando-daemon.toml.example`) says mt is a
`[[companion]]` to build, and `sando.toml` says which nodes install it.

The cost is that `Cargo.toml`'s version goes decorative, as mnw-cli's already
has. mt is not distributed on its own, so an independent version buys nothing.
Releases are identified by the bundle digest and the sha.

### Why the binary has no `static/` beside it

mt embeds its `static/` tree (`src/static_assets.rs`) instead of serving it from
a directory next to the binary. A companion is exactly one installed file, and
`build.rs` bakes a content hash of those assets into the templates compiled into
the binary — so a binary-only install would serve new markup against old CSS
with nothing reporting an error. Embedding makes the two inseparable.

Consequence for anyone editing assets: a change to `static/style.css`, `mt.js`
or the TypeScript under `frontend/src` needs a rebuild to take effect. It
already did, since those are what the `?v=` cache-buster hashes.

### One-time node setup

None outstanding. prod-1 has carried `/usr/local/lib/mnw/install-companion.sh`
and the `/etc/sudoers.d/mnw-companion` grant since mnw-cli, and the grant is on
the script rather than on a particular companion. testnot-1 got both on
2026-08-06, along with the postgres role and database, the system user,
`/opt/multithreaded/.env`, and the unit.

One testnot-only wrinkle worth recording, because it cost a debugging round and
will bite the next node set up the same way. testnot's `pg_hba.conf` has
`local all all peer map=mnw`, where prod has a plain `local all all peer`. A map
does **not** imply the identity pairing, so an OS user connecting as its own
same-named role is refused (`28000 Peer authentication failed`) until it is
listed in `pg_ident.conf`. `postgres:///multithreaded` therefore needed:

```
mnw             multithreaded           multithreaded
```

Any future node hosting mt needs both files, as root, from `sando/deploy/`:

```
sudo install -d /usr/local/lib/mnw
sudo install -m 0755 install-companion.sh /usr/local/lib/mnw/install-companion.sh
sudo install -m 0440 mnw-companion.sudoers /etc/sudoers.d/mnw-companion
sudo visudo -cf /etc/sudoers.d/mnw-companion
```

The sudoers file names every deploy user Sando SSHes as — `makenotwork` for
prod-1, `deploy` for testnot-1 — so it installs verbatim and needs no hand-edit.
A node Sando reaches as some other user needs that user added to the file, or the
install is denied, and denied late: companions install after the symlink swap.
Check the grant by hand before relying on it:

```
ssh <deploy-user>@<node> sudo /usr/local/lib/mnw/install-companion.sh
```

Exit 2 with the usage line means the grant works. A password prompt or a "not
allowed" means it does not.

mt also needs what it has always needed on the box, none of which Sando touches:
the `multithreaded` postgres role and database, `/opt/multithreaded/.env`,
`multithreaded.service` installed and enabled (`deploy/multithreaded.service`),
and a working OS trust store (see below).
Sando installs the binary and restarts the unit; it does not create users,
databases, env files, or units, and it will not rewrite a config that drifted.

### The OS trust store is a precondition

mt ships no trust anchors of its own. Every outbound TLS connection it makes
reads the host CA bundle, which on a Debian-family box means `/etc/ssl/certs`
kept current by the `ca-certificates` package. Three paths depend on it: the
OAuth token exchange with the MNW server (`src/auth.rs`), link previews
(`src/link_preview.rs`), and S3 or MinIO media through `s3-storage`. `reqwest`
reads the store through `rustls-platform-verifier` and the AWS client through
`rustls-native-certs`; neither offers bundled roots as an option any more, so
there is no in-binary fallback to fall back to.

A host with a thin or stale bundle does not fail in a way that looks like a
deploy problem. mt starts, `/api/health` passes because it only probes the
database, and pages serve. The first outbound TLS request is a login attempt,
because mt is an OAuth relying party, so the symptom is a login outage with a
certificate error buried in the journal on a node that reports healthy.

Two things now make that visible. mt probes the store at startup and logs an
error when it comes back empty, and `/api/health` carries a `tls_trust_anchors`
field that PoM asserts, so the condition surfaces as a degraded target rather
than as a support ticket. Neither watches the host bundle for staleness over
time; that is infra's job, and keeping `ca-certificates` on an update path is
still a precondition rather than something mt arranges.

Rollback is the MNW server's rollback: the previous bundle still holds the
previous mt binary.

## astra: the write-enabled harness instance

`deploy/deploy.sh` installs mt on astra. That instance is not a second staging
box in general; it exists so an audit run's browser lens can **write**.

The public demo (testnot) is deliberately read-only and no-login, which leaves
every mutation path browser-unverified, and mutation paths are where
rendered-behaviour bugs live: the CSRF token round-tripping through a real form
submission, HX swap targets, optimistic UI, the compiled islands once
`static/dist` is loaded. Verifying those needs a session, and a session needs a
real OAuth login. So the harness instance is reachable over the tailnet only,
logs in against testnot's MNW, and is wiped between runs.

Reached at `https://astra.tailc6b3e1.ts.net:3443`, published by `tailscale serve`
(`tailscale serve --bg --https=3443 http://127.0.0.1:3400`) and reachable from
the tailnet only. mt itself binds `127.0.0.1`, so the app port is on no other
interface and the tailnet proxy is the only way in.

TLS is not optional here, and finding that out is worth two minutes of someone
else's time: `config.rs` refuses to boot a non-loopback deployment on plain
`http`, because that would carry OAuth codes and session cookies in the clear.
`tailscale serve` is what supplies the certificate — astra's own Caddy already
holds `:443` for something else, hence the `:3443`. Set `COOKIE_SECURE=true`
to match.

### What it depends on, on the MNW side

Login is the whole point, so mt is only half the setup. The other half is
testnot's example seed, whose harness phase creates the accounts and the OAuth
client (`server/src/seed/harness.rs`):

| Piece | Value |
|-------|-------|
| `OAUTH_CLIENT_ID` | `mt-astra-harness` (a `sync_apps` row; PKCE, no secret) |
| Registered redirect | `https://astra.tailc6b3e1.ts.net:3443/auth/callback` |
| Accounts | `harness_fan` (Fan+), `harness_creator` (creator), `harness_owner` |
| Password | shared, from `MT_HARNESS_PASSWORD` on the testnot box |

That phase runs only when `MT_HARNESS_PASSWORD` and `MT_HARNESS_REDIRECT_URI`
are both set in testnot's `EnvironmentFile` (`/etc/mnw/makenotwork.env`), which
`mnw-testnot-seed.sh` already passes through. Neither value is in the repo: one
is a credential, the other names a tailnet host.

`SYNCKIT_JWT_SECRET` has to be set on that box too, and it was not until
2026-08-15. It is the HS256 key `/oauth/token` signs with, and without it every
token exchange answers 503 "SyncKit is not configured" — a login outage that
looks like a transport failure from mt's side, since mt retries a 5xx twice
before giving up. The seed does not set it (it is a machine secret, not seed
data), so a rebuilt testnot needs it added by hand: 32 characters minimum, or
the server refuses to boot.

The three accounts carry fixed MNW ids, and mt's own seed (`src/seed.rs`)
assigns their community roles by `mnw_account_id` — Owner and Moderator for
`harness_owner` — because login upserts identity and perks but never
membership. The two id lists have to be edited together; the MNW side has a test
pinning the literals.

### Walking it

```
deploy/harness-walk.sh
```

Logs in over OAuth as two harness accounts and walks the write paths the
instance exists to cover: thread create, reply, flag, and one moderation action,
each read back afterwards rather than trusted from its redirect. Run it after a
deploy to this box, and after any change to the login path on either side.

It is the check that covers what unit tests structurally cannot. The first run
of it found that mt had been sending the OAuth token exchange as JSON while the
server's `/oauth/token` takes `axum::Form`, so every login answered
`token_exchange_failed` on a 415 — in production too, since the server changed
extractors in April. mt's own tests passed throughout, because the wiremock
stand-in matched on path alone and answered any content type. The mocks now pin
the content type, and `callback_exchanges_the_code_as_a_form_and_logs_in` covers
the success path that had no test at all.

Credentials come from `~/.config/mnw/mt-harness.env` (mode 600, not in the
repo). Exit 0 means a browser lens can log in and write here.

### Resetting it

```
deploy/reset-astra.sh
```

Stops the service, drops every schema, re-migrates, re-seeds, and then asserts
the harness preconditions (three accounts, an Owner membership, threads
present). A run that logs in and finds no membership fails at the first write
with a 403, which reads like a permissions bug rather than an unfinished seed,
so the reset verifies rather than assuming.

This resets mt only. The identities live on testnot and come back from its own
reseed; the two are independent on purpose, because mt's state is disposable and
the accounts behind it are not.