max / makenotwork
| 1 | #!/usr/bin/env bash |
| 2 | # Rebuild and restart sandod itself to a target commit. |
| 3 | # |
| 4 | # Runs as ROOT, invoked by the oneshot unit `sando-update@<sha>.service` (the |
| 5 | # sha is the instance name, passed here as $1). sandod cannot do this itself: |
| 6 | # it runs User=sando with NoNewPrivileges + ProtectSystem=strict, so it can |
| 7 | # neither write /usr/local/bin/sandod nor restart its own service. sandod only |
| 8 | # *triggers* this unit (authorized for the sando user by a scoped polkit rule); |
| 9 | # the actual privileged work lives here. |
| 10 | # |
| 11 | # Build runs as the unprivileged build user (the sando user already carries a |
| 12 | # rustup toolchain at /srv/sando/.cargo/bin); only the install + restart run as |
| 13 | # root. The build uses a dedicated checkout, never the operator's dev tree. |
| 14 | # |
| 15 | # Config via environment (defaults shown), set in the unit or /etc/sando/sando.env: |
| 16 | # SANDO_SELF_UPDATE_DIR /srv/sando/self-update build checkout parent (build-user-owned) |
| 17 | # SANDO_UPSTREAM_URL /srv/sando/mnw.git source repo (see "Source" below) |
| 18 | # SANDO_BUILD_USER sando |
| 19 | # SANDO_BIN /usr/local/bin/sandod install destination |
| 20 | # SANDO_DEPLOY_BRANCH main the only branch a self-update sha may live on |
| 21 | # SANDO_DAEMON_CONFIG /etc/sando/sando-daemon.toml config the --check-config self-test loads |
| 22 | # SANDO_TOPOLOGY /etc/sando/sando.toml topology installed from the repo (see "Topology" below) |
| 23 | # |
| 24 | # Source: build from the LOCAL bare repo sandod already maintains |
| 25 | # (/srv/sando/mnw.git), not a remote fetch. Building from a remote |
| 26 | # (git@ssh.makenot.work) gave the `sando` user no git creds and broke the moment |
| 27 | # git hosting was down — it blocked the self-update during the 2026-07-09 deploy |
| 28 | # (postmortem #7). The local bare repo has no such external dependency. |
| 29 | # |
| 30 | # sandod fetches the deploy branch from the canonical remote into that bare repo |
| 31 | # at the top of its /self-update handler, so a sha pushed minutes ago is present |
| 32 | # here. It used to be reachable only if a *server* build had fetched since, which |
| 33 | # chained the controller's currency to the server's release cadence: shipping a |
| 34 | # controller fix meant cutting a server release first, and was impossible at all |
| 35 | # while the server was red. sandod remains the only writer of that repo, so the |
| 36 | # provenance seal below is unchanged. |
| 37 | # |
| 38 | # Provenance: sandod (bearer-gated) only *triggers* this unit with a hex sha; it |
| 39 | # does not prove the sha is a commit anyone intended to deploy. Without a check, |
| 40 | # the deploy token would be root code-exec on this host. So after fetch we REFUSE |
| 41 | # any sha that is not an ancestor of origin/$SANDO_DEPLOY_BRANCH: a feature-branch |
| 42 | # tip, an unknown sha, or a commit not on the deploy branch never reaches the |
| 43 | # build/install lines (exit 4). Only sandod writes the bare repo (fetching main |
| 44 | # from the authenticated upstream), so its main is a trustworthy provenance seal. |
| 45 | # A signed-tag check is the planned follow-up once release signing exists. |
| 46 | # |
| 47 | # Topology: this unit also INSTALLS `sando/sando.toml` from the checked-out sha |
| 48 | # over $SANDO_TOPOLOGY. The topology is not host-specific — it names tiers, |
| 49 | # nodes, gates and companions, and every one of those is a property of the |
| 50 | # deploy plan rather than of the box sandod happens to run on. It used to be |
| 51 | # hand-maintained, and the repo copy read as a source of truth that was |
| 52 | # deployed nowhere: the multithreaded companion block drifted for 19 days and |
| 53 | # was found by accident while diffing before an unrelated edit. Shipping it |
| 54 | # here makes the repo copy the deployed copy, which is the only arrangement |
| 55 | # where "edit the repo" and "change what sandod reads" are the same act. |
| 56 | # |
| 57 | # sandod cannot do this itself for the same reason it cannot install its own |
| 58 | # binary — and doing it from inside the daemon would be a bootstrap loop anyway, |
| 59 | # since the config being replaced is the one it is running on. The host-specific |
| 60 | # half stays where it was: $SANDO_DAEMON_CONFIG (bind address, tokens, database |
| 61 | # URL) is NOT installed from the repo, and `deploy/sando-daemon.toml.example` is |
| 62 | # still a template rather than a deployable file. |
| 63 | # |
| 64 | # The install lands BEFORE the --check-config self-test on purpose, so the test |
| 65 | # validates the new binary against the new topology — the pair that will |
| 66 | # actually boot. A failed test restores the previous topology before exiting, so |
| 67 | # a refused self-update leaves the box exactly as it found it. |
| 68 | # |
| 69 | # Safety net: a clean build (no stale incremental cache) plus a --check-config |
| 70 | # self-test of the freshly built binary against the LIVE config gate the install. |
| 71 | # A stale incremental object once produced a sandod that could not parse its own |
| 72 | # node_health config and crash-looped (postmortem #6); either guard alone stops |
| 73 | # that binary from ever being installed. |
| 74 | |
| 75 | |
| 76 | SHA="" |
| 77 | if ; then |
| 78 | |
| 79 | |
| 80 | fi |
| 81 | |
| 82 | SELF_DIR="" |
| 83 | UPSTREAM_URL="" |
| 84 | BUILD_USER="" |
| 85 | BIN="" |
| 86 | DEPLOY_BRANCH="" |
| 87 | DAEMON_CONFIG="" |
| 88 | TOPOLOGY="" |
| 89 | REPO_DIR="/MNW" |
| 90 | BUILD_HOME="" |
| 91 | |
| 92 | |
| 93 | |
| 94 | # Fetch + provenance check + checkout + build, all as the unprivileged build |
| 95 | # user. The clone is created once; thereafter we just fetch the new sha. Detached |
| 96 | # checkout so the dedicated tree never carries a branch to drift. |
| 97 | |
| 98 | |
| 99 | HOME="" \ |
| 100 | PATH="/.cargo/bin:/usr/local/bin:/usr/bin:/bin" \ |
| 101 | bash -euo pipefail -c " |
| 102 | if [[ ! -d '/.git' ]]; then |
| 103 | git clone '' '' |
| 104 | fi |
| 105 | cd '' |
| 106 | # Pin origin to the configured source every run, so switching |
| 107 | # SANDO_UPSTREAM_URL (e.g. remote -> local bare repo) takes effect on an |
| 108 | # already-cloned checkout instead of silently keeping the old remote. |
| 109 | git remote set-url origin '' |
| 110 | git fetch --prune origin |
| 111 | # Provenance seal: the sha must be reachable from the deploy branch in the |
| 112 | # source repo. --is-ancestor exits 1 for a non-ancestor and >1 for a |
| 113 | # bad/unresolvable ref, so any non-deploy-branch sha is refused fail-closed |
| 114 | # before a single line is built or installed. |
| 115 | if ! git merge-base --is-ancestor '' 'origin/'; then |
| 116 | echo \"sando-self-update: refusing sha '' — not an ancestor of origin/\" >&2 |
| 117 | exit 4 |
| 118 | fi |
| 119 | git checkout --detach '' |
| 120 | cd sando |
| 121 | # Clean build: wipe the workspace target so no stale incremental object |
| 122 | # survives across shas. A reused pre-node_health Gate enum object once |
| 123 | # produced a sandod that crash-looped on the current config (postmortem |
| 124 | # #6). Self-updates are rare, so a full recompile is a cheap insurance. |
| 125 | cargo clean |
| 126 | cargo build --release --locked -p sando-daemon |
| 127 | " |
| 128 | |
| 129 | # Workspace shares one target dir at sando/target; -p sando-daemon avoids |
| 130 | # compiling the TUI on the build host. |
| 131 | NEW_BIN="/sando/target/release/sandod" |
| 132 | || { ; ; } |
| 133 | |
| 134 | # Install the topology from the checked-out sha. This is what makes the repo copy |
| 135 | # the deployed copy; see "Topology" in the header. The daemon config beside it is |
| 136 | # host-specific and deliberately untouched. |
| 137 | # |
| 138 | # The previous copy is kept as a timestamped .bak so a bad topology is one `cp` |
| 139 | # away from being undone by hand, matching what the box already accumulates for |
| 140 | # every other file in /etc/sando. It is also what the self-test failure path |
| 141 | # below restores from. |
| 142 | NEW_TOPOLOGY="/sando/sando.toml" |
| 143 | || { ; ; } |
| 144 | TOPOLOGY_BACKUP="" |
| 145 | if ; then |
| 146 | if ; then |
| 147 | |
| 148 | else |
| 149 | TOPOLOGY_BACKUP=".bak-" |
| 150 | |
| 151 | |
| 152 | || |
| 153 | fi |
| 154 | fi |
| 155 | |
| 156 | |
| 157 | # Restore the previous topology on any failure from here to the restart. Without |
| 158 | # this, a refused self-update would leave the new topology in place under the old |
| 159 | # binary: sandod keeps running on what it parsed at boot, so the mismatch would |
| 160 | # surface at the next unrelated restart rather than here, which is the worst of |
| 161 | # both files. |
| 162 | |
| 163 | if ; then |
| 164 | |
| 165 | |
| 166 | fi |
| 167 | } |
| 168 | |
| 169 | # Self-test the fresh binary against the LIVE config BEFORE the swap: prove it can |
| 170 | # load + parse the exact daemon config + topology sandod will boot against. Run as |
| 171 | # the build user (not root) so the readability check matches the running daemon's |
| 172 | # identity. A binary that can't parse the current config (the postmortem #6 |
| 173 | # brick) fails here and is never installed — sandod keeps running on the old one. |
| 174 | # |
| 175 | # Since the topology was installed above, this now tests the pair: a topology the |
| 176 | # new binary cannot parse fails here too, and both halves are rolled back. |
| 177 | |
| 178 | if ! ; then |
| 179 | |
| 180 | |
| 181 | |
| 182 | fi |
| 183 | |
| 184 | # Install + restart as root. install is atomic (writes a temp then renames), so |
| 185 | # a concurrent exec of $BIN never sees a half-written file. |
| 186 | |
| 187 | |
| 188 | if ! ; then |
| 189 | |
| 190 | |
| 191 | |
| 192 | fi |
| 193 | |
| 194 |