max / makenotwork

1.3 KB · 23 lines History Blame Raw

1	-- Liveness heartbeat for the scan-job reaper.
2	--
3	-- Before this, `reap_stuck` reset any `running` job whose `started_at` was
4	-- older than STUCK_JOB_SECS (300s). But `started_at` is stamped at claim time
5	-- and spans the full S3 download (up to the spool ceiling) plus the scan, so a
6	-- legitimately slow large scan crossed the threshold while still running: a
7	-- second worker re-claimed it, both scanned the same object, and each claim
8	-- incremented `attempts` toward MAX_SCAN_ATTEMPTS -- force-retiring a valid file
9	-- to `failed` and stranding the entity at HeldForReview (audit Run 22).
10	--
11	-- The running worker now bumps `heartbeat_at` on a cadence well under
12	-- STUCK_JOB_SECS; the reaper and the PoM stuck-count key off
13	-- COALESCE(heartbeat_at, started_at), so only a job whose worker has actually
14	-- stopped touching it (a crashed or hung process) is reaped. A slow-but-
15	-- progressing scan keeps its heartbeat fresh and is never reclaimed. The COALESCE
16	-- fallback to started_at keeps pre-migration in-flight rows (heartbeat_at NULL)
17	-- reapable on the old clock.
18	ALTER TABLE scan_jobs ADD COLUMN IF NOT EXISTS heartbeat_at TIMESTAMPTZ;
19
20	-- No new index: the existing partial index `idx_scan_jobs_running`
21	-- (WHERE status = 'running') already bounds the reaper's candidate set to the
22	-- handful of in-flight rows, which it then filters by heartbeat.
23