max / makenotwork

Rebuild sando's tables with foreign keys off, not deferred Migration 011 failed at COMMIT on the live database with error 787 over a state that foreign_key_check calls consistent. defer_foreign_keys does not skip enforcement, it moves the check to COMMIT and counts outstanding violations meanwhile. DROP TABLE with foreign keys on performs an implicit DELETE of every parent row, and each child row pointing at one increments that counter. Nothing decrements it: the count falls only when DML resolves a violation, and re-creating the parent by DDL is not DML. It needs rows in the child tables to fire, and every test starts from an empty schema, so it could only ever fail in production -- 58 deploys and 418 gate_runs when the restart found it. PRAGMA foreign_keys is a no-op inside a transaction, so the migration carries -- no-transaction and opens its own. Same atomicity, and the pragma lands before the transaction begins. Editing an already-committed migration rather than adding a new one: 011 has never been applied anywhere. Prod is at 10, and the only other sando database in the tree has no _sqlx_migrations table at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-09 17:38 UTC

Signed with PGP, not checked

Commit: 60210607d8a22e67beca92b43c3e9ce7f2ad0f9d

Parent: 395681c

1 file changed, +38 insertions, -11 deletions

M sando/daemon/migrations/011_app_dimension.sql +38 -11

			@@ -1,3 +1,5 @@
	1	+	-- no-transaction
	2	+	--
1	3		-- Sando ships more than one product. Design: wiki [[sando-bento-boundary]].
2	4		--
3	5		-- Every table here was implicitly about the MNW server. `versions` keyed a
			@@ -21,19 +23,37 @@
21	23		-- anybody editing either.
22	24		--
23	25		-- Four tables need their PRIMARY KEY widened, which SQLite cannot do in place,
24		-	-- so they are rebuilt. `defer_foreign_keys` holds row-level enforcement until
25		-	-- COMMIT, which is what lets a referenced table be dropped and replaced with
26		-	-- rows still pointing at it. (`PRAGMA foreign_keys` itself is a no-op inside a
27		-	-- transaction, which is why it is not used here.)
	26	+	-- so they are rebuilt. That is the 12-step rebuild from the SQLite docs, and it
	27	+	-- wants `PRAGMA foreign_keys = OFF` around the whole thing.
28	28		--
29		-	-- Deferral does not cover everything, which decides the order below. A
30		-	-- composite `REFERENCES tiers(app, name)` is a schema error — "foreign key
31		-	-- mismatch" — until `tiers` actually has a unique index on those two columns,
32		-	-- and that is raised when the statement runs, not at COMMIT. So `tiers` is
33		-	-- rebuilt and renamed first, alone, and only then do the tables that reference
34		-	-- it get created.
	29	+	-- `defer_foreign_keys` is NOT a substitute, which is what this migration was
	30	+	-- written on and what broke it. Deferral does not skip enforcement, it moves
	31	+	-- the check to COMMIT and keeps a counter of outstanding violations in the
	32	+	-- meantime. With foreign keys on, `DROP TABLE` performs an implicit DELETE of
	33	+	-- every parent row, and each child row still pointing at one increments that
	34	+	-- counter. Nothing ever decrements it: the counter falls only when DML resolves
	35	+	-- a violation, and re-creating the parent by DDL is not DML. So COMMIT fails
	36	+	-- with error 787 over a database whose final state is consistent —
	37	+	-- `foreign_key_check` afterwards reports nothing.
	38	+	--
	39	+	-- The bug could only ever fire in production. It needs rows in the child
	40	+	-- tables, and every test starts from an empty schema. The live daemon had 58
	41	+	-- deploys and 418 gate_runs when this was found, on 2026-08-09, by the restart
	42	+	-- that was supposed to apply it.
	43	+	--
	44	+	-- `PRAGMA foreign_keys` is a no-op inside a transaction, so this migration
	45	+	-- carries `-- no-transaction` and opens its own instead. The atomicity is the
	46	+	-- same; what changes is that the pragma lands before the transaction begins.
	47	+	--
	48	+	-- The order below is unchanged and still load-bearing. A composite
	49	+	-- `REFERENCES tiers(app, name)` is a schema error — "foreign key mismatch" —
	50	+	-- until `tiers` actually has a unique index on those two columns, raised when
	51	+	-- the statement runs rather than at COMMIT, and no pragma turns that off. So
	52	+	-- `tiers` is rebuilt and renamed first, alone, and only then do the tables that
	53	+	-- reference it get created.
35	54
36		-	PRAGMA defer_foreign_keys = ON;
	55	+	PRAGMA foreign_keys = OFF;
	56	+	BEGIN;
37	57
38	58		-- Tiers are per product. MNW's `a`/`b` and pom's tiers are different pipelines
39	59		-- that happen to be spelled alike.
			@@ -187,3 +207,10 @@
187	207		CREATE INDEX gate_runs_lookup ON gate_runs(app, tier, version, gate_kind);
188	208		CREATE INDEX backups_name_fetched ON backups(app, name, fetched_at);
189	209		CREATE INDEX build_runs_by_sha ON build_runs(app, sha);
	210	+
	211	+	COMMIT;
	212	+
	213	+	-- Back on for the connection that ran this. The pragma is per connection rather
	214	+	-- than stored in the file, so leaving it off would only affect this one, but
	215	+	-- this one goes on to serve the daemon.
	216	+	PRAGMA foreign_keys = ON;