Skip to main content

max / makenotwork

pom: drop prod's remote CI check, and name the peers The check reported "tests: last test run failed" on mnw and mt from the minute it was added. It never ran. All three rows prod's test_runs table has ever held are exit 255 in zero seconds: hostkeys_foreach failed for /home/pom/.ssh/known_hosts: Permission denied Host key verification failed. It cannot be repaired in place. pom.service sets ProtectHome=true and ProtectSystem=strict, so the daemon cannot read or write /home/pom at all: no known_hosts, no identity, no ssh out of that host by any key. Nor should it have one -- an outbound identity there gives the public production box credentials into the build host, which is the wrong direction. CI already lives on astra, which runs the suites locally as the pom user and reports them: mnw 3337 passed, mt 500 passed, read 2026-08-23. Peer addresses go to tailnet names in both configs. One of them was already stale: [peers.macbook] pointed at 100.100.246.136, which is not in the tailnet at all -- mbp answers on 100.83.193.59 -- so that peer resolved to nothing, and on_missing = "log" is why nobody noticed. MagicDNS resolves astra from prod; verified both spellings answer identically before the change.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-23 18:22 UTC
Signed with PGP, not checked
Commit: d76122ae9895e0504eec2a3c549565931057eab6
Parent: 6885f1b
2 files changed, +48 insertions, -17 deletions
M pom/deploy/pom-astra.toml +8 -2
@@ -153,12 +153,18 @@
153 153 staleness_days = 7
154 154
155 155 [peers.hetzner]
156 - address = "100.120.174.96:9100"
156 + # By tailnet name. The label stays `hetzner` because that is what the MCP
157 + # tools address an instance by; only the address moves.
158 + address = "alpha-west-1:9100"
157 159 on_missing = "alert"
158 160 # token = "<hetzner's POM_API_TOKEN value>"
159 161
160 162 [peers.macbook]
161 - address = "100.100.246.136:9100"
163 + # `mbp`, not 100.100.246.136. That address was NOT IN THE TAILNET when this was
164 + # checked on 2026-08-23 -- mbp answers on 100.83.193.59 -- so this peer had been
165 + # pointing at nothing, quietly, for as long as nobody looked. `on_missing` is
166 + # "log" here, which is exactly why nobody did.
167 + address = "mbp:9100"
162 168 on_missing = "log"
163 169 # token = "<macbook's POM_API_TOKEN value>"
164 170
M pom/deploy/pom-hetzner.toml +40 -15
@@ -61,14 +61,38 @@
61 61 [targets.mnw.tls]
62 62 host = "makenot.work"
63 63
64 - [targets.mnw.tests]
65 - # Present in prod's live /etc/pom/pom.toml and previously missing here, which
66 - # made this file unsafe to deploy: installing it would have silently dropped the
67 - # remote CI check from prod. Keep the two in step.
68 - ssh = "max@100.106.221.39"
69 - command = "/home/max/staging/run-ci.sh"
70 - timeout_secs = 600
71 - staleness_days = 7
64 + # NO `[targets.mnw.tests]` HERE, and this is the correction rather than an
65 + # omission. Removed 2026-08-23 (infra `0db0a6a0`).
66 + #
67 + # What stood here was a remote CI check, `ssh = "max@100.106.221.39"` running
68 + # `/home/max/staging/run-ci.sh`, carried in this file because prod's live config
69 + # had it and the two were meant to stay in step. **It never once succeeded.**
70 + # Prod's `test_runs` table held three rows, all of them:
71 + #
72 + # exit_code 255, 0 seconds
73 + # hostkeys_find_by_key_hostfile: hostkeys_foreach failed for
74 + # /home/pom/.ssh/known_hosts: Permission denied
75 + # Host key verification failed.
76 + #
77 + # The first is dated 2026-08-20 22:55, which is when the check was added rather
78 + # than when it broke. It reported `tests: last test run failed` on both mnw and
79 + # mt from that minute onward.
80 + #
81 + # It cannot be repaired in place, which is why this is a removal and not a fixed
82 + # path. `pom.service` sets `ProtectHome=true` and `ProtectSystem=strict`, so the
83 + # daemon cannot read or write `/home/pom` at all: no `known_hosts`, no identity
84 + # file, no ssh out of this host by any key. That is the hardening working, not a
85 + # misconfiguration to relax.
86 + #
87 + # Nor should it be relaxed. An outbound SSH identity here would give the public
88 + # production box credentials into the build host, which is the wrong direction
89 + # for that trust to run.
90 + #
91 + # WHERE THE VERDICT LIVES: astra, which runs the suites locally as the `pom`
92 + # user (`/var/lib/pom/staging/run-ci.sh <target>`, see `pom-astra.toml`) and
93 + # reports them. On 2026-08-23 that instance read `mnw tests ok, 3337 passed` and
94 + # `mt tests ok, 500 passed`. Read it through the peer mesh; this instance
95 + # watches the platform, and CI belongs to the machine that can see the output.
72 96
73 97 [targets.mnw.backups]
74 98 # Where the backups actually are, verified on prod 2026-07-29: the nightly job
@@ -141,11 +165,8 @@
141 165 [targets.mt.tls]
142 166 host = "forums.makenot.work"
143 167
144 - [targets.mt.tests]
145 - ssh = "max@100.106.221.39"
146 - command = "cd /home/max/staging/multithreaded && cargo test --workspace 2>&1"
147 - timeout_secs = 300
148 - staleness_days = 7
168 + # No `[targets.mt.tests]` either, removed in the same pass and for the same
169 + # reason. The argument is written out under `[targets.mnw.tls]` above.
149 170
150 171 [targets.htpy]
151 172 label = "htpy.app"
@@ -177,12 +198,16 @@
177 198 host = "htpy.app"
178 199
179 200 [peers.astra]
180 - address = "100.106.221.39:9100"
201 + # By tailnet name. MagicDNS resolves it from this host: verified 2026-08-23,
202 + # `astra:9100` and the old literal both answered 401 identically.
203 + address = "astra:9100"
181 204 on_missing = "alert"
182 205 # token = "<astra's POM_API_TOKEN value>"
183 206
184 207 [peers.macbook]
185 - address = "100.100.246.136:9100"
208 + # See the note in `pom-astra.toml`: the literal that stood here was not in the
209 + # tailnet at all, so this peer resolved to nothing.
210 + address = "mbp:9100"
186 211 on_missing = "log"
187 212 # token = "<macbook's POM_API_TOKEN value>"
188 213