Skip to main content

max / makenotwork

Sandbox the makenotwork unit, on a measured write set ProtectSystem=strict plus the Protect*/Restrict* set, verified by running a probe under exactly these options as the service user on both prod and testnot: bare-repo creation, the scan spool, an export's private /tmp, clamd's unix socket, postgres over TCP and over its unix socket, and the yara rules all still work; /opt, /etc and the rest of /var are read-only. The memory ceiling is measured rather than copied. Prod's cgroup held 392M anonymous with a 403M peak, so the 512M this template wrote would have left ~110M of headroom and been an OOM kill on the first content export. 1G soft, 2G hard. Bootstrap now owns GIT_REPOS_PATH: it defaults to $STATE_DIR/git, gets created 2775 with group git so the service and the git user can both write, and is emitted as its own ReadWritePaths when an operator puts it outside STATE_DIR. The binary's own default is /opt/git, which the sandbox forbids. The directory permissions also fix a break that predates the sandbox: on prod GIT_REPOS_PATH is 0755 git:git, so the service cannot create a new owner directory and a creator's first repository fails. Prod still runs the old unit; applying it needs a restart in a deploy window.
Author: Max Johnson <me@maxj.phd> · 2026-08-25 13:09 UTC
Signed with PGP, not checked
Commit: 21eac73fa7b940b8e7e0f30b7e1ce59eb60dd5a1
Parent: 796ba5a
2 files changed, +91 insertions, -15 deletions
M sando/deploy/bootstrap-node.sh +68 -9
@@ -14,6 +14,13 @@
14 14 # bin_names[0]). Defaults to "makenotwork".
15 15 # SERVICE_NAME — systemd unit name. Defaults to "makenotwork.service".
16 16 # SERVICE_USER — runtime user for the binary. Defaults to "deploy".
17 + # GIT_REPOS_PATH — where the server keeps bare git repositories. Defaults
18 + # to $STATE_DIR/git, which is what prod and testnot both
19 + # run. It must match GIT_REPOS_PATH in the env file: the
20 + # unit's sandbox makes everything else read-only, and the
21 + # binary's own default is /opt/git, so a node that omits
22 + # it from the env file writes somewhere the sandbox
23 + # forbids.
17 24 # ENABLE_FIREWALL — "1" to set up UFW (22/80/443). Defaults to "1".
18 25 # INSTALL_CADDY — "1" to apt-install caddy (config is operator's job).
19 26 # Defaults to "1".
@@ -46,6 +53,7 @@
46 53 ETC_DIR="${ETC_DIR:-/etc/mnw}"
47 54 ENV_FILE="${ENV_FILE:-$ETC_DIR/makenotwork.env}"
48 55 STATE_DIR="${STATE_DIR:-/var/lib/mnw}"
56 + GIT_REPOS_PATH="${GIT_REPOS_PATH:-$STATE_DIR/git}"
49 57 BIN_NAME="${BIN_NAME:-makenotwork}"
50 58 SERVICE_NAME="${SERVICE_NAME:-makenotwork.service}"
51 59 SERVICE_USER="${SERVICE_USER:-deploy}"
@@ -130,6 +138,19 @@
130 138 fi
131 139 fi
132 140
141 + # Bare git repositories. On a host that runs git SSH, two accounts write here:
142 + # the web app creates the owner directory and the bare repo, and `git push`
143 + # writes objects as the git user. Group-writable + setgid so whichever of them
144 + # creates a directory, the other can still write inside it. Measured
145 + # 2026-08-25: prod has this directory 0755 git:git, so the service cannot
146 + # create a new owner directory at all and a creator's first repository fails
147 + # there today — unrelated to the sandbox, which is why it is fixed here.
148 + if getent passwd git >/dev/null; then
149 + install -d -o "$SERVICE_USER" -g git -m 2775 "$GIT_REPOS_PATH"
150 + else
151 + install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$GIT_REPOS_PATH"
152 + fi
153 +
133 154 log "6/8 sudoers (systemctl on $SERVICE_NAME for $SERVICE_USER)"
134 155 cat > "/etc/sudoers.d/${SERVICE_USER}-mnw" <<EOF
135 156 $SERVICE_USER ALL=(ALL) NOPASSWD: /bin/systemctl reload-or-restart $SERVICE_NAME, /bin/systemctl restart $SERVICE_NAME, /bin/systemctl status $SERVICE_NAME
@@ -155,7 +176,16 @@
155 176 EnvironmentFile=$ENV_FILE
156 177 # Runtime state (backups, spool, etc.) on FHS path; never inside the release
157 178 # dir or the deploy will erase it.
158 - ReadWritePaths=$STATE_DIR
179 + ReadWritePaths=$STATE_DIR$(
180 + # Bare repositories are normally under STATE_DIR and covered by the line
181 + # above. Emit a second path only when the operator has put them elsewhere,
182 + # so a sandbox that lists only STATE_DIR cannot silently break repository
183 + # creation.
184 + case "$GIT_REPOS_PATH/" in
185 + "$STATE_DIR"/*) ;;
186 + *) printf '\nReadWritePaths=%s' "$GIT_REPOS_PATH" ;;
187 + esac
188 + )
159 189 Restart=on-failure
160 190 RestartSec=30
161 191 # Exit 2 = migration failure (MNW server convention). Don't restart;
@@ -168,9 +198,44 @@
168 198 # rather than \$STATE_DIR and the two names are not interchangeable.
169 199 StateDirectory=makenotwork/scan-spool
170 200 StateDirectoryMode=0700
171 - # The ceilings server/docs/troubleshooting.md documents.
201 + # The ceilings server/docs/troubleshooting.md documents. Both are measured
202 + # rather than chosen: prod's own cgroup reported 392M anonymous (unreclaimable)
203 + # with a 403M peak on 2026-08-25, so the 512M this template used to write would
204 + # have left ~110M of headroom and been an OOM kill on the first content export,
205 + # which holds one file of up to 500M in memory. MemoryHigh throttles and
206 + # reclaims before MemoryMax kills. The soft NOFILE the service actually gets
207 + # without this line is 1024, not the 524288 that \`systemctl show\` reports
208 + # (that is the hard limit); the server had 18 descriptors open when measured.
172 209 LimitNOFILE=65535
173 - MemoryMax=512M
210 + MemoryHigh=1G
211 + MemoryMax=2G
212 + # The sandbox. Verified 2026-08-25 by running a probe under exactly these
213 + # options as the service user, on prod and on testnot: bare-repo creation, the
214 + # scan spool, an export's private /tmp, clamd's unix socket, postgres over both
215 + # TCP and its unix socket, and the yara rules all still work, while /opt, /etc
216 + # and the rest of /var are read-only. ProtectSystem=strict is what makes the
217 + # ReadWritePaths above load-bearing rather than decorative.
218 + NoNewPrivileges=yes
219 + ProtectSystem=strict
220 + ProtectHome=yes
221 + PrivateTmp=yes
222 + PrivateDevices=yes
223 + ProtectProc=invisible
224 + ProtectKernelTunables=yes
225 + ProtectKernelModules=yes
226 + ProtectKernelLogs=yes
227 + ProtectControlGroups=yes
228 + ProtectClock=yes
229 + ProtectHostname=yes
230 + RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
231 + RestrictNamespaces=yes
232 + RestrictRealtime=yes
233 + RestrictSUIDSGID=yes
234 + LockPersonality=yes
235 + MemoryDenyWriteExecute=yes
236 + SystemCallArchitectures=native
237 + SystemCallFilter=@system-service
238 + SystemCallErrorNumber=EPERM
174 239 StandardOutput=journal
175 240 StandardError=journal
176 241 SyslogIdentifier=$BIN_NAME
@@ -178,12 +243,6 @@
178 243 [Install]
179 244 WantedBy=multi-user.target
180 245 EOF
181 - # Deliberately absent: the systemd sandbox (ProtectSystem, NoNewPrivileges,
182 - # ProtectHome, the Restrict* set). Turning it on needs a measured
183 - # ReadWritePaths, because the server writes bare git repositories under
184 - # GIT_REPOS_PATH (/opt/git by default) as well as STATE_DIR and the scan
185 - # spool. A sandbox that lists only STATE_DIR breaks repository creation.
186 - # GoingsOn mnw-server task ad427fb3.
187 246 systemctl daemon-reload
188 247 systemctl enable "$SERVICE_NAME" >/dev/null 2>&1 || true
189 248
M server/docs/troubleshooting.md +23 -6
@@ -168,7 +168,7 @@
168 168 | Resource | Limit | What Happens |
169 169 |----------|-------|-------------|
170 170 | DB connections | 25 max | "timeout acquiring connection" after 3s wait |
171 - | Memory | 512M (systemd MemoryMax) | Process killed by OOM, auto-restarts |
171 + | Memory | 2G (systemd MemoryMax), throttled at 1G (MemoryHigh) | Process killed by OOM, auto-restarts |
172 172 | File descriptors | 65535 (LimitNOFILE) | "too many open files" |
173 173 | File upload: audio | 500 MB | 413 Payload Too Large |
174 174 | File upload: image | 10 MB | 413 Payload Too Large |
@@ -178,8 +178,25 @@
178 178 | SyncKit rate limit | 10/sec, burst 30 | 429 Too Many Requests |
179 179
180 180 The two systemd ceilings are what `sando/deploy/bootstrap-node.sh` writes into
181 - the unit. Prod was bootstrapped before they were in the template and reports
182 - `MemoryMax=infinity` and `LimitNOFILE=524288` as of 2026-08-22, so read the
183 - running unit with `systemctl show makenotwork -p MemoryMax -p LimitNOFILE`
184 - before treating either number as the reason for a symptom. GoingsOn mnw-server
185 - task `ad427fb3` closes the gap.
181 + the unit, along with the sandbox (`ProtectSystem=strict` plus the `Protect*` and
182 + `Restrict*` set). Prod was bootstrapped before any of it was in the template and
183 + still runs without it, so read the running unit before treating a ceiling as the
184 + reason for a symptom:
185 +
186 + ```bash
187 + systemctl show makenotwork -p MemoryMax -p MemoryHigh -p LimitNOFILE -p ProtectSystem
188 + ```
189 +
190 + The memory number is measured, not chosen. On 2026-08-25 prod's cgroup held 392M
191 + anonymous with a 403M peak, so the 512M this table used to claim would have been
192 + an OOM kill on the first content export (one file of up to 500M is in memory at a
193 + time). `LimitNOFILE` is the other trap: unset, the service gets a soft limit of
194 + 1024 while `systemctl show` reports the 524288 hard limit.
195 +
196 + Under the sandbox the only writable paths are `$STATE_DIR` (`/var/lib/mnw`,
197 + covering backups and the bare git repositories), the `StateDirectory` scan spool
198 + at `/var/lib/makenotwork/scan-spool`, and the unit's private `/tmp`. A permission
199 + error writing anywhere else is the sandbox, not a broken chown. `GIT_REPOS_PATH`
200 + in the env file has to stay inside `$STATE_DIR` or be listed as its own
201 + `ReadWritePaths`; the binary's own default is `/opt/git`, which the sandbox makes
202 + read-only.