max / makenotwork

Document the env-file mode invariant that broke a prod deploy The file must stay 0640 root:<service user>. It is easy to break and the breakage is invisible: systemd reads EnvironmentFile= as root, so the running service is fine, while Sando's pre-swap check reads it as the deploy user and the next deploy fails. The mechanism, from the 2026-07-31 write that caused it: rewriting through a temp file and `mv` replaces the inode, so the file inherits root's 077 umask and loses its ACL. Appending in place preserves the mode; `install -m 0640` sets it back when a rewrite is unavoidable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-01 16:58 UTC

Signed with PGP, not checked

Commit: c16feb25d531f2d4d59f5f2337c0108dc339596a

Parent: ef1bc43

1 file changed, +32 insertions, -0 deletions

M sando/deploy/README.md +32

			@@ -79,6 +79,38 @@
79	79		journalctl -u "sando-update@$SHA" -f
80	80		```
81	81
	82	+	## Editing a node's env file
	83	+
	84	+	`/etc/mnw/makenotwork.env` must stay mode 0640, owned root:<service user>.
	85	+	`bootstrap-node.sh` creates it that way and the deploy depends on it.
	86	+
	87	+	The trap, which cost a failed prod deploy on 2026-08-01: systemd reads
	88	+	`EnvironmentFile=` as root before dropping to `User=`, so the running service
	89	+	does not care about the mode. Sando's pre-swap config check is the only thing
	90	+	that reads the file as the deploy user, because its whole job is to load the
	91	+	config the way the service will before swapping the symlink. So a file rewritten
	92	+	0600 leaves production serving happily while the next deploy is already broken,
	93	+	and nothing says so until someone ships.
	94	+
	95	+	Rewriting the file through a temp file is what does it:
	96	+
	97	+	# WRONG: the temp file carries root's 077 umask, and `mv` carries it over
	98	+	{ cat /etc/mnw/makenotwork.env; echo "NEW_VAR=x"; } > /tmp/env && \
	99	+	mv /tmp/env /etc/mnw/makenotwork.env
	100	+
	101	+	`mv` replaces the inode, so the new file keeps the temp's 0600 and loses any ACL
	102	+	the old one carried. Append in place, or restore the mode explicitly afterwards:
	103	+
	104	+	# RIGHT: in-place, mode preserved
	105	+	echo "NEW_VAR=x" >> /etc/mnw/makenotwork.env
	106	+
	107	+	# or, if the file must be rewritten
	108	+	install -m 0640 -o root -g "$SERVICE_USER" /tmp/env /etc/mnw/makenotwork.env
	109	+
	110	+	The deploy now fails with a readability probe naming the user, the mode and the
	111	+	owner rather than a bare `Permission denied` from inside a generated script, so
	112	+	this is recoverable in one read — but it is still a failed deploy.
	113	+
82	114		## Rollback contract
83	115
84	116		A Sando rollback (canary rollback of a node, or an operator swapping the