max / makenotwork

2.0 KB · 34 lines History Blame Raw

1	-- Record when a tier's deployed identity (current_version / current_build_id)
2	-- last changed, so a startup reconcile can catch a promote that died between
3	-- landing on the nodes and advancing tier_state.
4	--
5	-- The gap this closes (audit 2026-07-21, sando-h5-crash-mid-promote): a promote
6	-- inserts the per-node `deploys` row (outcome 'ok') and then calls advance_tier.
7	-- If sandod is SIGKILLed between those two writes (OOM, systemctl restart, a
8	-- self-update), the node is running the new version but tier_state still records
9	-- the old one, and partial_reason is NULL because the failure path never ran.
10	-- /state reports a clean tier on the wrong version, and a later /rollback then
11	-- targets previous_version -- now two versions behind what the node is running.
12	--
13	-- The reconcile needs a trustworthy "tier_state last changed at" instant to
14	-- decide whether a `deploys` row postdates the recorded state. burn_in_started_at
15	-- cannot serve that role: /rollback and reset_burn_in both null it while the
16	-- deployed identity is real, so a NULL there is ambiguous. advanced_at is set on
17	-- every genuine advance (runs::advance_tier) and every rollback (routes) and is
18	-- never nulled, so `deploy.finished_at > advanced_at` cleanly means "a deploy
19	-- landed after the last state change we recorded".
20	ALTER TABLE tier_state ADD COLUMN advanced_at TEXT;
21
22	-- Backfill existing rows to trust current state: seed advanced_at from the
23	-- burn-in clock when present, else from the tier's most recent deploy. This
24	-- makes the first post-migration boot quiet across every existing tier
25	-- (clean, rolled-back, or burn-in-reset) -- the reconcile only fires on a
26	-- crash that happens after this column starts being maintained live. A tier
27	-- with neither signal (never deployed) keeps NULL and has no deploys row for
28	-- the reconcile to weigh, so it is skipped anyway.
29	UPDATE tier_state
30	SET advanced_at = COALESCE(
31	burn_in_started_at,
32	(SELECT MAX(d.finished_at) FROM deploys d WHERE d.tier = tier_state.tier)
33	);
34