| 1 |
# SSH access: production server (Hetzner) |
| 2 |
|
| 3 |
Two SSH paths into the production server, with different audiences and |
| 4 |
different break-glass behavior. Read this before disabling either one. |
| 5 |
|
| 6 |
## The two paths |
| 7 |
|
| 8 |
|
| 9 |
|
| 10 |
| `22` | public (eth0) | mnw-cli git operations | anywhere (firewall + sshd config) | |
| 11 |
| `2200` | Tailscale (tailscale0) | admin access | tailnet only (firewall blocks public) | |
| 12 |
|
| 13 |
- **Public :22** is intentionally open so creators can `git clone`/`push` |
| 14 |
over SSH against `ssh.makenot.work`. The sshd config on this port is |
| 15 |
locked to git-shell only. See `setup-git-ssh.sh` and `sshd-git.conf`. |
| 16 |
No interactive shell, no port forwarding, no admin access. |
| 17 |
- **Tailnet :2200** is the admin path. Full interactive shell, used for |
| 18 |
every Sando deploy and any manual maintenance. Reachable only |
| 19 |
from devices on the tailnet (firewall rule `ufw allow in on tailscale0` |
| 20 |
in `setup-firewall.sh`). |
| 21 |
|
| 22 |
## Why this split exists |
| 23 |
|
| 24 |
Tailscale runs its own SSH server when configured. If that goes down |
| 25 |
(Tailscale service crashes, ACL misconfiguration, accidental |
| 26 |
`tailscale down`) and public sshd is also locked down, admin access is |
| 27 |
gone entirely. |
| 28 |
|
| 29 |
The split prevents that: public :22 is always alive (firewalled to allow |
| 30 |
SSH from anywhere) but restricted to git-shell, so an attacker who |
| 31 |
probes :22 finds nothing but git commands. Admin :2200 lives on the |
| 32 |
tailnet, where the firewall blocks public access and the surface area |
| 33 |
is small. |
| 34 |
|
| 35 |
## Break-glass procedure |
| 36 |
|
| 37 |
If the tailnet path stops working (Tailscale down, key revoked, ACL |
| 38 |
broken): |
| 39 |
|
| 40 |
1. **Verify public sshd is up** from any machine: |
| 41 |
`ssh -p 22 root@5.78.144.244 -o BatchMode=yes -o ConnectTimeout=5 true` |
| 42 |
Expect a key-based prompt or a refused git-shell. Both prove sshd |
| 43 |
is listening. A timeout or "Connection refused" means public sshd is |
| 44 |
ALSO down and you need Hetzner Cloud Console. |
| 45 |
2. **Edit `/etc/ssh/sshd_config.d/git-shell.conf` from Hetzner Console** |
| 46 |
to temporarily restore an interactive shell for the `root` user on |
| 47 |
port `22`. Match `User root` block, set `ForceCommand` to nothing. |
| 48 |
3. **Restart sshd**: `systemctl restart ssh`. Test from your laptop. |
| 49 |
4. **Fix the tailnet path** (re-auth `tailscale up`, restore key, etc). |
| 50 |
5. **Revert the sshd edit** and restart `ssh` again. Don't leave the |
| 51 |
interactive root shell on public :22. It defeats the whole split. |
| 52 |
|
| 53 |
## What NOT to do |
| 54 |
|
| 55 |
- **Do not** disable Tailscale SSH (`tailscale set --ssh=false`) without |
| 56 |
first proving public :22 is reachable and you have a working root key |
| 57 |
for it. Memory rule: `feedback_tailscale_ssh`, getting locked out |
| 58 |
requires Hetzner Console access, which costs time we don't always |
| 59 |
have. |
| 60 |
- **Do not** restrict public :22 to specific IPs without coordinating: |
| 61 |
the mnw-cli git endpoint serves users worldwide. |
| 62 |
- **Do not** open port 2200 to the public. It's the admin shell. |
| 63 |
|
| 64 |
## Hetzner Console (last resort) |
| 65 |
|
| 66 |
If both paths are dead, the Hetzner Cloud Console provides KVM-style |
| 67 |
access to the server's serial console. Login at |
| 68 |
<https://console.hetzner.cloud>, select the project, select the server, |
| 69 |
"Console" tab. Slow but always available. Same root credentials. |
| 70 |
|
| 71 |
## Key paths |
| 72 |
|
| 73 |
- Firewall: `deploy/setup-firewall.sh` |
| 74 |
- Public sshd (git-only): `deploy/sshd-git.conf`, `deploy/setup-git-ssh.sh` |
| 75 |
- Admin sshd (tailnet): `/etc/ssh/sshd_config` on the server |
| 76 |
- Deploy entry point: Sando (`sandod` on fw13, uses `-p 2200`). The retired |
| 77 |
hand-deploy script is `deploy/archive/deploy.sh.legacy`. |
| 78 |
- CI runner setup: `deploy/setup-ci.sh` (uses `-p 2200`) |
| 79 |
|