Skip to main content

max / makenotwork

Deploy the Sando topology from the repo, and report it when it drifts /etc/sando/sando.toml was installed once at bootstrap and hand-edited after, while MNW/sando/sando.toml read as the source of truth and was deployed nowhere. Nothing kept them in step: the multithreaded companion block drifted for 19 days and was found by accident while diffing before an unrelated edit. sando-self-update.sh now installs the topology from the checked-out sha, before its --check-config self-test, so the test validates the binary and topology that boot together. A failed test or restart restores the previous copy, kept as a timestamped .bak. sandod cannot do this itself under ProtectSystem=strict, and replacing the config it is running on from inside itself would be a bootstrap loop regardless. check-topology-drift.sh plus sando-config-drift.timer cover the gap between self-updates: an unshipped commit on main, or a hand edit to the live file. It reads the bare repo sandod maintains rather than the operator's dev tree, and reports rather than installing. A failed unit is the signal. The daemon config beside it stays host-specific and hand-held, and secrets in sando.env are untouched by any of this.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-28 14:48 UTC
Signed with PGP, not checked
Commit: bc1aa36059b76b5114fcb3e594345f284d37d935
Parent: 74c6dd1
7 files changed, +220 insertions, -2 deletions
M sando/sando.toml +6
@@ -8,6 +8,12 @@
8 8 # declared but not provisioned; adding the second prod node later is a config
9 9 # edit (set provisioned = true, fill in [[tier.node]]).
10 10 #
11 + # THIS FILE IS DEPLOYED, NOT A DOCUMENT. `sando-self-update.sh` installs it over
12 + # /etc/sando/sando.toml on every controller self-update, and
13 + # `sando-config-drift.timer` reports it daily when the two separate. Change what
14 + # sandod reads by editing here and shipping; do not hand-edit the installed copy.
15 + # See sando/deploy/README.md, "The topology is deployed, not hand-maintained".
16 + #
11 17 # The first tier is "host" — it refers to whatever machine sandod runs on
12 18 # (currently fw13). Renamed from the legacy "mm" name in Session 1 of
13 19 # the sando bundle redesign.
M sando/deploy/README.md +34 -1
@@ -11,7 +11,7 @@
11 11 | `sandod.service` | Sando host | The Sando daemon (`sandod`). |
12 12 | `bootstrap-sandod-host.sh` | Sando host | One-time host setup for the daemon. |
13 13 | `bootstrap-node.sh` | a deploy target | One-time node setup (release dirs, deploy user, service). |
14 - | `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando.toml`). |
14 + | `sando-daemon.toml.example` | Sando host | Template for the daemon config (`sando-daemon.toml`). Host-specific; not deployed from the repo. |
15 15 | `post-receive` | git remote | Push-to-deploy hook. |
16 16 | `sandod-backup-fetch.{service,timer}` | Sando host | Daily pull of every configured prod dump into `/srv/sando/backups/` (04:00 UTC). |
17 17 | `mnw-testnot-seed.{sh,service}` | Sando host | Reset testnot.work to the fabricated example catalog (`--seed-examples`). On-demand, not scheduled. Ends by running the smoke check below. |
@@ -19,8 +19,41 @@
19 19 | `sando-update@.service` + `sando-self-update.sh` | Sando host | Self-update: rebuild + restart `sandod` to a target sha. |
20 20 | `10-sando-update.rules` | Sando host | polkit grant letting the `sando` user start (only) `sando-update@*`. |
21 21 | `install-companion.sh` | a deploy target | Install a staged companion binary + restart its unit (companion services). |
22 + | `check-topology-drift.sh` | Sando host | Diff `main:sando/sando.toml` against the live `/etc/sando/sando.toml`; non-zero on drift. |
23 + | `sando-config-drift.{service,timer}` | Sando host | Daily run of the check above (05:00 UTC). A failed unit is the drift signal. |
22 24 | `mnw-companion.sudoers` | a deploy target | Scoped sudo grant for the deploy user to run `install-companion.sh`. |
23 25
26 + ## The topology is deployed, not hand-maintained
27 +
28 + `sando.toml` (the topology: tiers, nodes, gates, companions) is installed from
29 + the repo. Bootstrap installs it, and `sando-self-update.sh` reinstalls it from
30 + the checked-out sha on every self-update, so the way to change what `sandod`
31 + reads is to edit `MNW/sando/sando.toml` and ship it.
32 +
33 + This is not how it started. The repo copy read as a source of truth and was
34 + deployed nowhere, `/etc/sando/sando.toml` was hand-edited, and nothing kept the
35 + two in step: the multithreaded companion block drifted for 19 days and was found
36 + by accident while diffing before an unrelated edit.
37 +
38 + Three things hold it together now:
39 +
40 + - The self-update installs the topology **before** its `--check-config`
41 + self-test, so the test validates the binary and topology that will boot
42 + together. A failed test or a failed restart restores the previous topology, so
43 + a refused self-update leaves the box as it found it. The replaced copy is kept
44 + as `sando.toml.bak-<timestamp>`.
45 + - `sando-config-drift.timer` reports the gap that opens between self-updates: a
46 + commit on main nobody has shipped yet, or a hand edit to the live file. It
47 + reads and reports, never installs.
48 + - The daemon config beside it, `sando-daemon.toml`, stays host-specific and
49 + hand-held: bind address, tokens, database URLs. `sando-daemon.toml.example` is
50 + a template, not a deployable file. Secrets live in `sando.env` (0640
51 + root:sando) and are untouched by any of this.
52 +
53 + `sandod` does none of the installing itself. It cannot write `/etc/sando` under
54 + `ProtectSystem=strict`, and replacing the config it is running on from inside
55 + itself would be a bootstrap loop regardless.
56 +
24 57 ## Self-update (deploying the controller itself)
25 58
26 59 The deploy pipeline ships the *MNW server*, not `sandod`. To deploy a new
M sando/deploy/bootstrap-sandod-host.sh +18
@@ -215,6 +215,11 @@
215 215 install -m 0644 -o root -g root \
216 216 "$SCRIPT_DIR/sando-daemon.toml.example" \
217 217 /etc/sando/sando-daemon.toml
218 + # The topology. Bootstrap is not the only writer any more: `sando-self-update.sh`
219 + # reinstalls this file from the checked-out sha on every self-update, so the repo
220 + # copy stays the deployed copy instead of separating from it between bootstraps.
221 + # Do not hand-edit the installed file; `sando-config-drift.timer` below reports
222 + # it when someone does.
218 223 install -m 0644 -o root -g root \
219 224 "$SANDO_REPO/sando.toml" \
220 225 /etc/sando/sando.toml
@@ -239,6 +244,12 @@
239 244 chmod 0640 /etc/sando/sando.env
240 245
241 246 log "11/13 systemd units"
247 + # The drift check reads the bare repo and the live topology, so it goes beside
248 + # the units that run it rather than in /usr/local/bin with the daemon.
249 + install -d -m 0755 /usr/local/lib/sando
250 + install -m 0755 -o root -g root \
251 + "$SCRIPT_DIR/check-topology-drift.sh" \
252 + /usr/local/lib/sando/check-topology-drift.sh
242 253 install -m 0644 -o root -g root \
243 254 "$SCRIPT_DIR/sandod.service" \
244 255 /etc/systemd/system/sandod.service
@@ -248,6 +259,12 @@
248 259 install -m 0644 -o root -g root \
249 260 "$SCRIPT_DIR/sandod-backup-fetch.timer" \
250 261 /etc/systemd/system/sandod-backup-fetch.timer
262 + install -m 0644 -o root -g root \
263 + "$SCRIPT_DIR/sando-config-drift.service" \
264 + /etc/systemd/system/sando-config-drift.service
265 + install -m 0644 -o root -g root \
266 + "$SCRIPT_DIR/sando-config-drift.timer" \
267 + /etc/systemd/system/sando-config-drift.timer
251 268 systemctl daemon-reload
252 269
253 270 if [[ "$BUILD_SANDOD" == "1" ]]; then
@@ -283,6 +300,7 @@
283 300 # to start against an incomplete environment.
284 301 systemctl enable sandod.service >/dev/null 2>&1 || true
285 302 systemctl enable sandod-backup-fetch.timer >/dev/null 2>&1 || true
303 + systemctl enable sando-config-drift.timer >/dev/null 2>&1 || true
286 304
287 305 echo
288 306 log "Done. Next steps for the operator:"
M sando/deploy/sando-self-update.sh +68 -1
@@ -19,6 +19,7 @@
19 19 # SANDO_BIN /usr/local/bin/sandod install destination
20 20 # SANDO_DEPLOY_BRANCH main the only branch a self-update sha may live on
21 21 # SANDO_DAEMON_CONFIG /etc/sando/sando-daemon.toml config the --check-config self-test loads
22 + # SANDO_TOPOLOGY /etc/sando/sando.toml topology installed from the repo (see "Topology" below)
22 23 #
23 24 # Source: build from the LOCAL bare repo sandod already maintains
24 25 # (/srv/sando/mnw.git), not a remote fetch. Building from a remote
@@ -43,6 +44,28 @@
43 44 # from the authenticated upstream), so its main is a trustworthy provenance seal.
44 45 # A signed-tag check is the planned follow-up once release signing exists.
45 46 #
47 + # Topology: this unit also INSTALLS `sando/sando.toml` from the checked-out sha
48 + # over $SANDO_TOPOLOGY. The topology is not host-specific — it names tiers,
49 + # nodes, gates and companions, and every one of those is a property of the
50 + # deploy plan rather than of the box sandod happens to run on. It used to be
51 + # hand-maintained, and the repo copy read as a source of truth that was
52 + # deployed nowhere: the multithreaded companion block drifted for 19 days and
53 + # was found by accident while diffing before an unrelated edit. Shipping it
54 + # here makes the repo copy the deployed copy, which is the only arrangement
55 + # where "edit the repo" and "change what sandod reads" are the same act.
56 + #
57 + # sandod cannot do this itself for the same reason it cannot install its own
58 + # binary — and doing it from inside the daemon would be a bootstrap loop anyway,
59 + # since the config being replaced is the one it is running on. The host-specific
60 + # half stays where it was: $SANDO_DAEMON_CONFIG (bind address, tokens, database
61 + # URL) is NOT installed from the repo, and `deploy/sando-daemon.toml.example` is
62 + # still a template rather than a deployable file.
63 + #
64 + # The install lands BEFORE the --check-config self-test on purpose, so the test
65 + # validates the new binary against the new topology — the pair that will
66 + # actually boot. A failed test restores the previous topology before exiting, so
67 + # a refused self-update leaves the box exactly as it found it.
68 + #
46 69 # Safety net: a clean build (no stale incremental cache) plus a --check-config
47 70 # self-test of the freshly built binary against the LIVE config gate the install.
48 71 # A stale incremental object once produced a sandod that could not parse its own
@@ -62,6 +85,7 @@
62 85 BIN="${SANDO_BIN:-/usr/local/bin/sandod}"
63 86 DEPLOY_BRANCH="${SANDO_DEPLOY_BRANCH:-main}"
64 87 DAEMON_CONFIG="${SANDO_DAEMON_CONFIG:-/etc/sando/sando-daemon.toml}"
88 + TOPOLOGY="${SANDO_TOPOLOGY:-/etc/sando/sando.toml}"
65 89 REPO_DIR="$SELF_DIR/MNW"
66 90 BUILD_HOME="$(getent passwd "$BUILD_USER" | cut -d: -f6)"
67 91
@@ -107,14 +131,53 @@
107 131 NEW_BIN="$REPO_DIR/sando/target/release/sandod"
108 132 [[ -x "$NEW_BIN" ]] || { echo "sando-self-update: build produced no binary at $NEW_BIN" >&2; exit 3; }
109 133
134 + # Install the topology from the checked-out sha. This is what makes the repo copy
135 + # the deployed copy; see "Topology" in the header. The daemon config beside it is
136 + # host-specific and deliberately untouched.
137 + #
138 + # The previous copy is kept as a timestamped .bak so a bad topology is one `cp`
139 + # away from being undone by hand, matching what the box already accumulates for
140 + # every other file in /etc/sando. It is also what the self-test failure path
141 + # below restores from.
142 + NEW_TOPOLOGY="$REPO_DIR/sando/sando.toml"
143 + [[ -f "$NEW_TOPOLOGY" ]] || { echo "sando-self-update: no topology at $NEW_TOPOLOGY in sha $SHA" >&2; exit 6; }
144 + TOPOLOGY_BACKUP=""
145 + if [[ -f "$TOPOLOGY" ]]; then
146 + if cmp -s "$NEW_TOPOLOGY" "$TOPOLOGY"; then
147 + echo "sando-self-update: topology unchanged"
148 + else
149 + TOPOLOGY_BACKUP="$TOPOLOGY.bak-$(date -u +%Y%m%dT%H%M%SZ)"
150 + cp -p "$TOPOLOGY" "$TOPOLOGY_BACKUP"
151 + echo "sando-self-update: topology differs; previous copy saved to $TOPOLOGY_BACKUP"
152 + diff -u "$TOPOLOGY_BACKUP" "$NEW_TOPOLOGY" || true
153 + fi
154 + fi
155 + install -m 0644 "$NEW_TOPOLOGY" "$TOPOLOGY"
156 +
157 + # Restore the previous topology on any failure from here to the restart. Without
158 + # this, a refused self-update would leave the new topology in place under the old
159 + # binary: sandod keeps running on what it parsed at boot, so the mismatch would
160 + # surface at the next unrelated restart rather than here, which is the worst of
161 + # both files.
162 + restore_topology() {
163 + if [[ -n "$TOPOLOGY_BACKUP" && -f "$TOPOLOGY_BACKUP" ]]; then
164 + install -m 0644 "$TOPOLOGY_BACKUP" "$TOPOLOGY"
165 + echo "sando-self-update: restored the previous topology from $TOPOLOGY_BACKUP" >&2
166 + fi
167 + }
168 +
110 169 # Self-test the fresh binary against the LIVE config BEFORE the swap: prove it can
111 170 # load + parse the exact daemon config + topology sandod will boot against. Run as
112 171 # the build user (not root) so the readability check matches the running daemon's
113 172 # identity. A binary that can't parse the current config (the postmortem #6
114 173 # brick) fails here and is never installed — sandod keeps running on the old one.
174 + #
175 + # Since the topology was installed above, this now tests the pair: a topology the
176 + # new binary cannot parse fails here too, and both halves are rolled back.
115 177 echo "sando-self-update: self-testing $NEW_BIN against $DAEMON_CONFIG"
116 178 if ! runuser -u "$BUILD_USER" -- env SANDO_CONFIG="$DAEMON_CONFIG" "$NEW_BIN" --check-config; then
117 179 echo "sando-self-update: new binary FAILED --check-config against $DAEMON_CONFIG; refusing to install (sandod left running on the current binary)" >&2
180 + restore_topology
118 181 exit 5
119 182 fi
120 183
@@ -122,5 +185,9 @@
122 185 # a concurrent exec of $BIN never sees a half-written file.
123 186 echo "sando-self-update: installing $NEW_BIN -> $BIN and restarting sandod"
124 187 install -m 0755 "$NEW_BIN" "$BIN"
125 - systemctl restart sandod
188 + if ! systemctl restart sandod; then
189 + echo "sando-self-update: sandod failed to restart on the new binary + topology" >&2
190 + restore_topology
191 + exit 7
192 + fi
126 193 echo "sando-self-update: done ($SHA live)"
A sando/deploy/check-topology-drift.sh +53
@@ -1,0 +1,53 @@
1 + #!/usr/bin/env bash
2 + # Fail when the live Sando topology is not what the repo says it should be.
3 + #
4 + # The self-update oneshot installs `sando/sando.toml` over /etc/sando/sando.toml,
5 + # so between self-updates the two can still separate: a commit lands on main and
6 + # nothing has shipped it yet, or somebody edits the live file by hand. Neither is
7 + # visible without someone remembering to diff, which is exactly how the
8 + # multithreaded companion block drifted for 19 days.
9 + #
10 + # This is the reporting half. It changes nothing; it exits non-zero and prints
11 + # the diff. Run from `sando-config-drift.timer` daily, where a non-zero exit
12 + # leaves a failed unit for PoM to see, or by hand on the Sando host.
13 + #
14 + # Source of truth is the deploy branch of the bare repo sandod maintains, NOT the
15 + # operator's dev tree: an uncommitted edit in ~/Code is not deployable and should
16 + # not read as drift, and the bare repo is the same provenance seal the
17 + # self-update's ancestor check trusts.
18 + #
19 + # Config via environment (defaults shown):
20 + # SANDO_UPSTREAM_URL /srv/sando/mnw.git bare repo to read the topology from
21 + # SANDO_DEPLOY_BRANCH main branch within it
22 + # SANDO_TOPOLOGY /etc/sando/sando.toml the live file
23 + set -euo pipefail
24 +
25 + BARE="${SANDO_UPSTREAM_URL:-/srv/sando/mnw.git}"
26 + BRANCH="${SANDO_DEPLOY_BRANCH:-main}"
27 + TOPOLOGY="${SANDO_TOPOLOGY:-/etc/sando/sando.toml}"
28 +
29 + [[ -f "$TOPOLOGY" ]] || { echo "check-topology-drift: no live topology at $TOPOLOGY" >&2; exit 2; }
30 +
31 + # No fetch. sandod is the only writer of this repo and refreshes it on every
32 + # build and self-update; fetching here would need git credentials this unit
33 + # deliberately does not carry, and a stale bare repo is the self-update's
34 + # problem rather than a drift report's.
35 + if ! REPO_COPY="$(git -C "$BARE" show "$BRANCH:sando/sando.toml" 2>/dev/null)"; then
36 + echo "check-topology-drift: cannot read $BRANCH:sando/sando.toml from $BARE" >&2
37 + exit 2
38 + fi
39 +
40 + if diff -u --label "$BARE ($BRANCH)" --label "$TOPOLOGY" <(printf '%s\n' "$REPO_COPY") "$TOPOLOGY"; then
41 + echo "check-topology-drift: live topology matches $BRANCH"
42 + exit 0
43 + fi
44 +
45 + cat >&2 <<'MSG'
46 +
47 + check-topology-drift: the live topology and the repo have separated.
48 +
49 + Fix it by shipping, not by editing the live file: push the intended topology to
50 + main and trigger a sandod self-update, which installs it. Hand-editing
51 + /etc/sando/sando.toml is what this check exists to catch.
52 + MSG
53 + exit 1
A sando/deploy/sando-config-drift.service +21
@@ -1,0 +1,21 @@
1 + # One-shot: report when the live Sando topology is not what main says.
2 + #
3 + # Place at /etc/systemd/system/sando-config-drift.service on the Sando host.
4 + #
5 + # Reads only — it never installs a topology. A non-zero exit leaves this unit in
6 + # `failed`, which is the point: that is the loud, standing signal PoM's systemd
7 + # checks already look at, and it costs nothing to keep armed.
8 +
9 + [Unit]
10 + Description=Sando: report topology drift between the repo and the live config
11 + After=sandod.service
12 + # Deliberately not Requires=: a stopped sandod does not make the live file any
13 + # less drifted, and this check reads files rather than talking to the daemon.
14 +
15 + [Service]
16 + Type=oneshot
17 + ExecStart=/usr/local/lib/sando/check-topology-drift.sh
18 + # The bare repo is sando-owned and the live topology is world-readable, so this
19 + # needs no privilege at all.
20 + User=sando
21 + Group=sando
A sando/deploy/sando-config-drift.timer +20
@@ -1,0 +1,20 @@
1 + # Daily trigger for the topology drift check.
2 + #
3 + # Place at /etc/systemd/system/sando-config-drift.timer on the Sando host.
4 + # Enable: systemctl enable --now sando-config-drift.timer
5 + #
6 + # 05:00 UTC, an hour after the backup fetch, so the two never contend for the
7 + # same wakeup and their logs stay easy to read apart.
8 +
9 + [Unit]
10 + Description=Sando: daily topology drift check
11 +
12 + [Timer]
13 + OnCalendar=*-*-* 05:00:00 UTC
14 + # If the box was off when the timer fired, run on next boot. Drift is a standing
15 + # condition rather than a moment, so a late run is still the right answer.
16 + Persistent=true
17 + Unit=sando-config-drift.service
18 +
19 + [Install]
20 + WantedBy=timers.target