Skip to main content

max / makenotwork

server: run scheduler payments/cleanup I/O off the tick lock The webhook-retry, refund/credit-escalation, credit settle/reverse, and stale-transaction/orphaned-upload cleanup ran serially inline in the tick — up to ~310 serial Stripe/S3 round-trips (30-90s) while the scheduler held its single-instance advisory lock, tripping the 50s WAM tick-overrun alert and threatening to eat the 60s interval (fuzz-2026-07-06 F2). Spawn the block on the background pool (matching the existing release-announcements offload) so the lock releases as soon as the DB-only jobs finish. Every operation here is already safe without the tick lock — per-event advisory-lock dedup, FOR UPDATE SKIP LOCKED claims, and deterministic Stripe idempotency keys (the code already relies on next-tick retry) — so a run overlapping the next tick just skips locked rows. Internal ordering is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-07 15:22 UTC
Commit: 7b639a657c78bf8fd8a7bc767f31c4fab614cd2a
Parent: 5db553a
1 file changed, +25 insertions, -20 deletions
@@ -241,26 +241,31 @@ pub fn spawn_scheduler(
241 241 }
242 242 }
243 243
244 - // Retry failed webhook events
245 - webhooks::retry_failed_webhooks(&state).await;
246 -
247 - // Escalate stale pending refunds (unmatched for >24 hours)
248 - webhooks::escalate_stale_refunds(&state).await;
249 -
250 - // Settle owed platform-funded credits (Fan+ reimbursements) via transfer,
251 - // then escalate any that were claimed but never completed (>24h).
252 - webhooks::settle_platform_credits(&state).await;
253 - webhooks::escalate_stale_platform_credits(&state).await;
254 -
255 - // Reverse settled platform credits whose sale was later refunded, so
256 - // MNW isn't left funding a returned item (Run 21 money-loss finding).
257 - webhooks::reverse_refunded_platform_credits(&state).await;
258 -
259 - // Clean up stale pending transactions (>24h) and release promo code reservations
260 - cleanup::cleanup_stale_pending_transactions(&state).await;
261 -
262 - // Delete S3 objects from presigned uploads that were never confirmed (>24h)
263 - cleanup::cleanup_orphaned_uploads(&state).await;
244 + // Payments + storage outbound I/O: webhook retry, refund/credit
245 + // escalation, platform-credit settle/reverse, stale-transaction and
246 + // orphaned-upload cleanup. This block issues up to ~310 serial
247 + // Stripe/S3 round-trips (30-90s worst case). Run it on the background
248 + // pool rather than inline so the scheduler advisory lock is released as
249 + // soon as the DB-only jobs finish instead of being held for the full
250 + // outbound-I/O duration (fuzz-2026-07-06 F2 — the hold tripped the 50s
251 + // WAM tick-overrun alert and could eat the 60s interval). Every
252 + // operation here is already safe to run without the single-instance
253 + // tick lock — per-event `pg_try_advisory_xact_lock` dedup, `FOR UPDATE
254 + // SKIP LOCKED` claims, and deterministic Stripe idempotency keys — so a
255 + // run overlapping the next tick's just skips locked rows. Internal
256 + // ordering (settle before escalate) is preserved by the sequence here.
257 + {
258 + let job_state = state.clone();
259 + state.bg.spawn("scheduler-payments-io", async move {
260 + webhooks::retry_failed_webhooks(&job_state).await;
261 + webhooks::escalate_stale_refunds(&job_state).await;
262 + webhooks::settle_platform_credits(&job_state).await;
263 + webhooks::escalate_stale_platform_credits(&job_state).await;
264 + webhooks::reverse_refunded_platform_credits(&job_state).await;
265 + cleanup::cleanup_stale_pending_transactions(&job_state).await;
266 + cleanup::cleanup_orphaned_uploads(&job_state).await;
267 + });
268 + }
264 269
265 270 // Hourly: scan SyncKit apps for 75/90/100% cap breaches and email
266 271 // the app owner. Cheap query — single JOIN on a small table.