Skip to main content

max / makenotwork

server: rewrite scan pipeline (async + dashboard + 5 new layers) Implements Phases 1-3 of docs/scan-pipeline-audit.md as a single coherent rewrite of the malware-scan pipeline. Triggered by the MalwareBazaar regression that silently held every upload since 2026-05-10 because of the global fail-closed-by-default policy. Architecture (Phase 1) - Per-layer ERROR_POLICY const co-located in each layer file (FailClosed for in-process deterministic layers; FailOpen for external network layers). final_status aggregator consults each layer's declared policy via error_policy_for. Removes the global has-error switch that caused the regression. - Async scan_jobs queue (migration 121) drained by a pool of tokio workers (constants::SCAN_WORKER_COUNT = 2) using FOR UPDATE SKIP LOCKED. Upload routes enqueue and return Pending immediately; worker flips the target entity's scan_status when done. Stuck-job reaper resets jobs older than 5 minutes. - scan_and_classify is replaced by enqueue_scan_for; all 7 callsites migrated. Trusted-user fast-path preserved; quarantine-then-bail semantics replaced by admin dashboard cleanup. - FileScanStatus::Scanning added. Admin dashboard (Phase 2) - /admin/uploads expanded to three-section layout (Pipeline Health panel, Active Queue counts, Held for Review table with per-layer chips) plus collapsible Recent History grid and full audit log at /admin/uploads/audit. - scan_admin_actions table (migration 122) records every promote, quarantine, and rescan. Each held row carries an inline "last: promote by max 2m ago" tooltip via batched DISTINCT ON lookups. - Per-row actions: Promote, Quarantine, Re-scan. Bulk operation: Re-scan all held (the operational fix for the existing 9-version backlog after deploy). - HTMX polls /admin/uploads/queue-summary every 10s for live pending+running counts. - CSS for layer chips (verdict color-coded), layer health cards (status-badge left border), scan-history grid. Layer-set expansion (Phase 3) - malwarebazaar: adds Auth-Key header for the abuse.ch API change, short-circuits to Skip without a key, explicit handling for unauthorized / key_required / key_invalid status values. - urlhaus: new layer. Pure-Rust URL extraction from binary strings, dedupes hosts, queries up to 5 against URLhaus with same key. - signing_macos: new layer (apple-codesign). Mach-O / .app / DMG verification with team-id extraction and notarization staple presence detection. Powered by indygreg's apple-codesign 0.29.0. - signing_windows: new layer (authenticode). PE detection, Authenticode signature parse, subject-CN extraction. - signing_linux: new layer. AppImage detection via ELF + AI marker; presence check for .sha256_sig + .sig_key sections. - metadefender: new layer. Second-opinion only — invoked when another layer flagged the upload suspicious (suspicion_present gating). Free-tier 40/day. Fail-open per policy. - Config: URLHAUS_ENABLED, ABUSE_CH_AUTH_KEY, METADEFENDER_API_KEY env vars wired through ScanConfig. Trade-offs - Quarantine is no longer rejected synchronously. The upload completes, scan runs async, admin handles cleanup via dashboard. - Project images and content_insertions enqueue but have no entity scan_status column — worker only records file_scan_results and tickets quarantine via WAM. - Notarization staple verification is presence + structural sanity; cryptographic CMS chain verification against Apple's notarization CA is deferred to a hardening pass. Tests: 1622 passing, including 21 new policy + suspicion + classify tests + per-layer verdict tests for each of the 5 new layers.
Co-Authored-By
Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author: Max J. <87768334+MaxJMath@users.noreply.github.com> · 2026-05-24 20:54 UTC
Commit: 34a68b7bc9683da41eaa2564c1a1af99206bd876
Parent: 0d02c2f
41 files changed, +3764 insertions, -238 deletions
@@ -352,6 +352,16 @@
352 352 pub yara_rules_dir: String,
353 353 /// Whether to enable MalwareBazaar hash lookups
354 354 pub malwarebazaar_enabled: bool,
355 + /// Whether to enable URLhaus URL-reputation lookups
356 + pub urlhaus_enabled: bool,
357 + /// Shared abuse.ch Auth-Key (issued at https://auth.abuse.ch/). Required
358 + /// for MalwareBazaar and URLhaus as of 2024+; without it both layers
359 + /// fail-open and the dashboard surfaces them as degraded.
360 + pub abuse_ch_auth_key: Option<String>,
361 + /// MetaDefender Cloud API key (free tier at
362 + /// <https://metadefender.com/account>). Second-opinion layer; only
363 + /// invoked when another layer flagged the file as suspicious.
364 + pub metadefender_api_key: Option<String>,
355 365 }
356 366
357 367 impl ScanConfig {
@@ -373,6 +383,11 @@
373 383 malwarebazaar_enabled: std::env::var("MALWAREBAZAAR_ENABLED")
374 384 .map(|v| v != "false" && v != "0")
375 385 .unwrap_or(true),
386 + urlhaus_enabled: std::env::var("URLHAUS_ENABLED")
387 + .map(|v| v != "false" && v != "0")
388 + .unwrap_or(true),
389 + abuse_ch_auth_key: std::env::var("ABUSE_CH_AUTH_KEY").ok().filter(|s| !s.is_empty()),
390 + metadefender_api_key: std::env::var("METADEFENDER_API_KEY").ok().filter(|s| !s.is_empty()),
376 391 })
377 392 }
378 393 }
@@ -383,6 +398,9 @@
383 398 .field("clamav_socket", &self.clamav_socket)
384 399 .field("yara_rules_dir", &self.yara_rules_dir)
385 400 .field("malwarebazaar_enabled", &self.malwarebazaar_enabled)
401 + .field("urlhaus_enabled", &self.urlhaus_enabled)
402 + .field("abuse_ch_auth_key", &self.abuse_ch_auth_key.as_ref().map(|_| "<set>"))
403 + .field("metadefender_api_key", &self.metadefender_api_key.as_ref().map(|_| "<set>"))
386 404 .finish()
387 405 }
388 406 }
@@ -517,7 +535,8 @@
517 535 "SYNCKIT_S3_SECRET_KEY", "SYNCKIT_S3_REGION",
518 536 "STRIPE_SECRET_KEY", "STRIPE_WEBHOOK_SECRET", "STRIPE_WEBHOOK_SECRET_V2",
519 537 "ADMIN_USER_ID", "SYNCKIT_JWT_SECRET", "SCAN_ENABLED", "CLAMAV_SOCKET",
520 - "YARA_RULES_DIR", "MALWAREBAZAAR_ENABLED", "GIT_REPOS_PATH",
538 + "YARA_RULES_DIR", "MALWAREBAZAAR_ENABLED", "URLHAUS_ENABLED",
539 + "ABUSE_CH_AUTH_KEY", "METADEFENDER_API_KEY", "GIT_REPOS_PATH",
521 540 "POSTMARK_WEBHOOK_TOKEN", "POSTMARK_BROADCAST_WEBHOOK_TOKEN",
522 541 "GIT_SSH_HOST", "MT_BASE_URL", "FAN_PLUS_STRIPE_PRICE_ID",
523 542 "CREATOR_TIER_BASIC_PRICE_ID", "CREATOR_TIER_SMALL_FILES_PRICE_ID",
@@ -145,6 +145,7 @@
145 145 // -- File scanning --
146 146 pub const SCAN_MAX_MEMORY_BYTES: usize = 100 * 1024 * 1024; // 100 MB in-memory threshold
147 147 pub const SCAN_MAX_CONCURRENT: usize = 4; // Max concurrent file scans (each can use up to 100 MB RAM)
148 + pub const SCAN_WORKER_COUNT: usize = 2; // Background worker tasks draining scan_jobs queue
148 149
149 150 // -- Caddy on-demand TLS --
150 151 // Caps concurrent cache-miss DB lookups in `/api/domains/caddy-ask`. Cache hits
@@ -327,6 +327,22 @@
327 327 let scheduler_shutdown_rx = shutdown_tx.subscribe();
328 328 let _scheduler_handle = makenotwork::scheduler::spawn_scheduler(state.clone(), scheduler_shutdown_rx);
329 329
330 + // Start scan worker pool. Only meaningful if a scanner is configured;
331 + // otherwise enqueue_scan_for never enqueues (trust-gate fast path).
332 + if let (Some(scanner), Some(s3_for_workers)) = (state.scanner.clone(), state.s3.clone()) {
333 + let scan_ctx = std::sync::Arc::new(makenotwork::scanning::worker::WorkerContext {
334 + db: state.db.clone(),
335 + s3: s3_for_workers,
336 + pipeline: scanner,
337 + scan_semaphore: state.scan_semaphore.clone(),
338 + wam: state.wam.clone(),
339 + });
340 + let worker_count = makenotwork::constants::SCAN_WORKER_COUNT;
341 + let worker_shutdown_rx = shutdown_tx.subscribe();
342 + makenotwork::scanning::worker::spawn_pool(worker_count, scan_ctx, worker_shutdown_rx);
343 + tracing::info!(worker_count, "scan worker pool started");
344 + }
345 +
330 346 // Build router (shared with integration tests via lib.rs)
331 347 let app = build_app(state, session_layer)
332 348 // Request ID: propagate → trace → set (Axum applies inside-out)
@@ -3962,6 +3962,99 @@
3962 3962 font-family: var(--font-mono);
3963 3963 }
3964 3964
3965 + /* Scan pipeline dashboard. See docs/scan-pipeline-audit.md § 5. */
3966 +
3967 + .admin-scan-section {
3968 + margin-top: var(--space-6);
3969 + }
3970 +
3971 + .admin-section-header {
3972 + display: flex;
3973 + justify-content: space-between;
3974 + align-items: baseline;
3975 + margin-bottom: var(--space-3);
3976 + gap: var(--space-3);
3977 + }
3978 +
3979 + .admin-section-header h2 {
3980 + font-size: 1.1rem;
3981 + font-family: var(--font-mono);
3982 + margin: 0;
3983 + }
3984 +
3985 + .layer-health-grid {
3986 + display: grid;
3987 + grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
3988 + gap: var(--space-3);
3989 + }
3990 +
3991 + .layer-health-card {
3992 + padding: var(--space-3);
3993 + border: 1px solid var(--border);
3994 + border-radius: var(--radius-md);
3995 + border-left-width: 4px;
3996 + font-size: 0.85rem;
3997 + }
3998 +
3999 + .layer-health-card .layer-name {
4000 + font-family: var(--font-mono);
4001 + font-weight: 600;
4002 + font-size: 0.95rem;
4003 + margin-bottom: var(--space-2);
4004 + }
4005 +
4006 + .layer-health-card .layer-stat {
4007 + display: flex;
4008 + justify-content: space-between;
4009 + margin-bottom: 2px;
4010 + }
4011 +
4012 + .layer-health-card .layer-stat .dimmed {
4013 + color: var(--text-muted);
4014 + }
4015 +
4016 + .layer-health-card.layer-ok { border-left-color: var(--success, #527a3e); }
4017 + .layer-health-card.layer-degraded { border-left-color: var(--warning); }
4018 + .layer-health-card.layer-down { border-left-color: var(--danger); }
4019 +
4020 + .layer-chips {
4021 + display: flex;
4022 + flex-wrap: wrap;
4023 + gap: 4px;
4024 + max-width: 360px;
4025 + }
4026 +
4027 + .layer-chip {
4028 + display: inline-block;
4029 + font-family: var(--font-mono);
4030 + font-size: 0.7rem;
4031 + padding: 1px 6px;
4032 + border-radius: var(--radius-sm);
4033 + border: 1px solid transparent;
4034 + cursor: help;
4035 + line-height: 1.4;
4036 + }
4037 +
4038 + .layer-chip-pass { background: #e6f0dc; border-color: #c4d8a8; color: #3a5a2a; }
4039 + .layer-chip-skip { background: var(--bg-page); border-color: var(--border); color: var(--text-muted); }
4040 + .layer-chip-fail { background: var(--danger-bg); border-color: var(--danger); color: var(--danger); font-weight: 600; }
4041 + .layer-chip-error { background: var(--warning-bg); border-color: var(--warning-border); color: var(--warning); font-weight: 600; }
4042 +
4043 + .scan-history-grid table { font-size: 0.85rem; }
4044 +
4045 + .scan-history-grid summary {
4046 + cursor: pointer;
4047 + font-family: var(--font-mono);
4048 + padding: var(--space-2) 0;
4049 + user-select: none;
4050 + }
4051 +
4052 + .scan-row-meta {
4053 + font-size: 0.75rem;
4054 + color: var(--text-muted);
4055 + margin-top: 2px;
4056 + }
4057 +
3965 4058 .admin-page table {
3966 4059 width: 100%;
3967 4060 border-collapse: collapse;
@@ -277,10 +277,22 @@
277 277
278 278 // ── File scanning ──
279 279
280 + /// Status of an uploaded file in the scan pipeline.
281 + ///
282 + /// `Pending` — accepted, waiting in `scan_jobs` queue for a worker.
283 + /// `Scanning` — worker has claimed the job and is running the pipeline.
284 + /// `Clean` — pipeline completed, no Fail verdicts, no fail-closed Errors.
285 + /// `HeldForReview` — pipeline completed with a fail-closed Error, OR the
286 + /// uploader is untrusted (every untrusted upload routes to admin review).
287 + /// `Quarantined` — pipeline returned a Fail verdict on at least one layer.
288 + /// `Error` — pipeline itself crashed (worker exception, S3 fetch failed, etc.).
289 + ///
290 + /// State machine in `docs/scan-pipeline-audit.md`.
280 291 #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
281 292 #[serde(rename_all = "snake_case")]
282 293 pub enum FileScanStatus {
283 294 Pending,
295 + Scanning,
284 296 Clean,
285 297 Quarantined,
286 298 HeldForReview,
@@ -289,6 +301,7 @@
289 301
290 302 impl_str_enum!(FileScanStatus {
291 303 Pending => "pending",
304 + Scanning => "scanning",
292 305 Clean => "clean",
293 306 Quarantined => "quarantined",
294 307 HeldForReview => "held_for_review",
@@ -1068,6 +1081,10 @@
1068 1081 #[test]
1069 1082 fn file_scan_status_round_trip() {
1070 1083 assert_eq!(FileScanStatus::Clean.to_string(), "clean");
1084 + assert_eq!(FileScanStatus::Pending.to_string(), "pending");
1085 + assert_eq!(FileScanStatus::Scanning.to_string(), "scanning");
1086 + assert_eq!("pending".parse::<FileScanStatus>().unwrap(), FileScanStatus::Pending);
1087 + assert_eq!("scanning".parse::<FileScanStatus>().unwrap(), FileScanStatus::Scanning);
1071 1088 assert_eq!("held_for_review".parse::<FileScanStatus>().unwrap(), FileScanStatus::HeldForReview);
1072 1089 assert_eq!(FileScanStatus::HeldForReview.to_string(), "held_for_review");
1073 1090 assert_eq!("quarantined".parse::<FileScanStatus>().unwrap(), FileScanStatus::Quarantined);
@@ -36,6 +36,8 @@
36 36 pub(crate) mod health;
37 37 pub(crate) mod monitor;
38 38 pub(crate) mod scanning;
39 + pub(crate) mod scan_jobs;
40 + pub(crate) mod scan_admin_actions;
39 41 pub(crate) mod content_insertions;
40 42 pub(crate) mod invites;
41 43 pub(crate) mod analytics;
@@ -10,7 +10,7 @@
10 10 use super::VersionId;
11 11 use crate::scanning::ScanResult;
12 12
13 - /// An item held for review, joined with creator info.
13 + /// An item held for review, joined with creator info and latest scan layers.
14 14 #[derive(Debug, Clone, FromRow)]
15 15 pub struct HeldItemRow {
16 16 pub item_id: ItemId,
@@ -20,9 +20,12 @@
20 20 pub creator_id: UserId,
21 21 pub upload_trusted: bool,
22 22 pub held_at: DateTime<Utc>,
23 + /// Latest `file_scan_results.scan_layers` JSON for this entity's s3_key,
24 + /// or `null` if no scan has run yet. The dashboard renders this as chips.
25 + pub scan_layers: Option<serde_json::Value>,
23 26 }
24 27
25 - /// A version held for review, joined with creator info.
28 + /// A version held for review, joined with creator info and latest scan layers.
26 29 #[derive(Debug, Clone, FromRow)]
27 30 pub struct HeldVersionRow {
28 31 pub item_id: ItemId,
@@ -34,6 +37,7 @@
34 37 pub creator_id: UserId,
35 38 pub upload_trusted: bool,
36 39 pub held_at: DateTime<Utc>,
40 + pub scan_layers: Option<serde_json::Value>,
37 41 }
38 42
39 43 /// Insert a scan result record for audit trail.
@@ -104,7 +108,8 @@
104 108 Ok(())
105 109 }
106 110
107 - /// Get items held for review, joined with creator info. Oldest first.
111 + /// Get items held for review, joined with creator info + latest scan layers.
112 + /// Oldest first.
108 113 #[tracing::instrument(skip_all)]
109 114 pub async fn get_held_items(db: &PgPool) -> Result<Vec<HeldItemRow>, sqlx::Error> {
110 115 let rows = sqlx::query_as::<_, HeldItemRow>(
@@ -112,7 +117,12 @@
112 117 SELECT i.id AS item_id, i.title AS item_title,
113 118 COALESCE(i.audio_s3_key, i.cover_s3_key) AS s3_key,
114 119 u.username AS creator_username, u.id AS creator_id,
115 - u.upload_trusted, i.updated_at AS held_at
120 + u.upload_trusted, i.updated_at AS held_at,
121 + (
122 + SELECT fsr.scan_layers FROM file_scan_results fsr
123 + WHERE fsr.s3_key = COALESCE(i.audio_s3_key, i.cover_s3_key)
124 + ORDER BY fsr.scanned_at DESC LIMIT 1
125 + ) AS scan_layers
116 126 FROM items i
117 127 JOIN projects p ON p.id = i.project_id
118 128 JOIN users u ON u.id = p.user_id
@@ -126,7 +136,8 @@
126 136 Ok(rows)
127 137 }
128 138
129 - /// Get versions held for review, joined with creator info. Oldest first.
139 + /// Get versions held for review, joined with creator info + latest scan layers.
140 + /// Oldest first.
130 141 #[tracing::instrument(skip_all)]
131 142 pub async fn get_held_versions(db: &PgPool) -> Result<Vec<HeldVersionRow>, sqlx::Error> {
132 143 let rows = sqlx::query_as::<_, HeldVersionRow>(
@@ -135,7 +146,12 @@
135 146 v.id AS version_id, v.version_number,
136 147 v.s3_key,
137 148 u.username AS creator_username, u.id AS creator_id,
138 - u.upload_trusted, v.created_at AS held_at
149 + u.upload_trusted, v.created_at AS held_at,
150 + (
151 + SELECT fsr.scan_layers FROM file_scan_results fsr
152 + WHERE fsr.s3_key = v.s3_key
153 + ORDER BY fsr.scanned_at DESC LIMIT 1
154 + ) AS scan_layers
139 155 FROM versions v
140 156 JOIN items i ON i.id = v.item_id
141 157 JOIN projects p ON p.id = i.project_id
@@ -149,3 +165,87 @@
149 165
150 166 Ok(rows)
151 167 }
168 +
169 + /// Per-layer aggregate stats over a window for the admin dashboard.
170 + #[derive(Debug, Clone, FromRow)]
171 + pub struct LayerHealthRow {
172 + pub layer: String,
173 + pub pass_count: i64,
174 + pub skip_count: i64,
175 + pub fail_count: i64,
176 + pub error_count: i64,
177 + pub last_pass_or_skip: Option<DateTime<Utc>>,
178 + }
179 +
180 + /// Compute per-layer health stats over the last N hours.
181 + ///
182 + /// Reads `file_scan_results.scan_layers` JSONB and rolls up verdict counts
183 + /// per layer. `last_pass_or_skip` is the most recent timestamp at which the
184 + /// layer returned a non-error, non-fail verdict — the indicator the admin
185 + /// panel uses to flag a layer as down.
186 + #[tracing::instrument(skip_all)]
187 + pub async fn layer_health_window(
188 + db: &PgPool,
189 + hours: i64,
190 + ) -> Result<Vec<LayerHealthRow>, sqlx::Error> {
191 + sqlx::query_as::<_, LayerHealthRow>(
192 + r#"
193 + WITH expanded AS (
194 + SELECT fsr.scanned_at,
195 + (l ->> 'layer') AS layer,
196 + (l ->> 'verdict') AS verdict
197 + FROM file_scan_results fsr,
198 + jsonb_array_elements(fsr.scan_layers) AS l
199 + WHERE fsr.scanned_at > NOW() - ($1 || ' hours')::interval
200 + )
201 + SELECT layer,
202 + COUNT(*) FILTER (WHERE verdict = 'pass') AS pass_count,
203 + COUNT(*) FILTER (WHERE verdict = 'skip') AS skip_count,
204 + COUNT(*) FILTER (WHERE verdict = 'fail') AS fail_count,
205 + COUNT(*) FILTER (WHERE verdict = 'error') AS error_count,
206 + MAX(scanned_at) FILTER (WHERE verdict IN ('pass', 'skip')) AS last_pass_or_skip
207 + FROM expanded
208 + GROUP BY layer
209 + ORDER BY layer
210 + "#,
211 + )
212 + .bind(hours.to_string())
213 + .fetch_all(db)
214 + .await
215 + }
216 +
217 + /// A scan-history row for the dashboard's "Recent" grid.
218 + #[derive(Debug, Clone, FromRow)]
219 + pub struct ScanHistoryRow {
220 + pub scanned_at: DateTime<Utc>,
221 + pub s3_key: String,
222 + pub scan_status: String,
223 + pub sha256: Option<String>,
224 + pub file_size_bytes: Option<i64>,
225 + pub scan_layers: serde_json::Value,
226 + }
227 +
228 + /// Recent scan results across all entities. Newest first, capped at `limit`.
229 + /// Used by the Recent History collapsible section. `since_hours` bounds the
230 + /// window so the grid renders fast.
231 + #[tracing::instrument(skip_all)]
232 + pub async fn recent_history(
233 + db: &PgPool,
234 + since_hours: i64,
235 + limit: i64,
236 + ) -> Result<Vec<ScanHistoryRow>, sqlx::Error> {
237 + sqlx::query_as::<_, ScanHistoryRow>(
238 + r#"
239 + SELECT fsr.scanned_at, fsr.s3_key, fsr.scan_status,
240 + fsr.sha256, fsr.file_size_bytes, fsr.scan_layers
241 + FROM file_scan_results fsr
242 + WHERE fsr.scanned_at > NOW() - ($1 || ' hours')::interval
243 + ORDER BY fsr.scanned_at DESC
244 + LIMIT $2
245 + "#,
246 + )
247 + .bind(since_hours.to_string())
248 + .bind(limit)
249 + .fetch_all(db)
250 + .await
251 + }