Skip to main content

max / makenotwork

Alert on security signals instead of accumulating them The 2026-07-21 audit collapsed sixty findings into one root cause: nothing tells you when something breaks. CSP reporting was that bug at page level, and the last violation was found by a human opening a browser console. 5xx spikes, credential stuffing, rate-limit volume and webhook signature failures were the same bug one layer up: logged, and a log nobody reads is not a control. Reuses what is already built rather than adding a monitoring product. Counters live in process, in fixed windows, and cross a threshold at most once per window; a firing lands in admin_alerts and mails ALERT_EMAIL, the same path POST /api/internal/alerts already had. That endpoint exists for external agents, so in-process signals call insert_alert directly instead of posting to ourselves over loopback with a bearer token, and the mail half moves into a shared email_alert so both paths produce the same row and the same mail. ALERTS_INGEST_TOKEN gates inbound requests from other machines and is not a precondition for any of this. AlertKind gains one Security variant, not five event-shaped ones. The module's own rule is that kinds are domain-level and the sub-condition rides in dedup_key, the way Tls already folds three PoM categories. Nothing dedups server-side: insert_alert is a plain INSERT and there is no unique constraint on dedup_key. The once-per-window rule is therefore the only thing between one condition and a mailbox full of identical alerts, so the test that pins it is load-bearing. Auth failures are counted at the failure sites, not inferred from status: a wrong password re-renders the login form with a 200 and a rejected passkey assertion leaves as a 400, so nothing about either response says an attempt failed. CSP alerts only on an explicitly enforced violation, because the Report-Only policy shipped alongside the enforced one fires on ordinary traffic by design. Delivery is best effort throughout: it logs on failure, never blocks a request, and never panics. An alerting channel that can take the site down is worse than no alerting channel.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-01 15:16 UTC
Signed with PGP, not checked
Commit: 263201c54541365ba8fbb9995d0dfb6b39ee2b16
Parent: 4235ff7
12 files changed, +681 insertions, -35 deletions
M server/Cargo.lock +1 -1
@@ -2579,7 +2579,7 @@
2579 2579
2580 2580 [[package]]
2581 2581 name = "docengine"
2582 - version = "0.3.5"
2582 + version = "0.4.0"
2583 2583 dependencies = [
2584 2584 "ammonia",
2585 2585 "pulldown-cmark",
M server/src/lib.rs +1
@@ -51,6 +51,7 @@
51 51 pub mod rss;
52 52 pub mod scanning;
53 53 pub mod scheduler;
54 + pub mod security_signals;
54 55 pub mod seed;
55 56 pub mod storage;
56 57 pub mod synckit_auth;
M server/src/main.rs +27
@@ -584,8 +584,35 @@
584 584 }
585 585 }
586 586
587 + // Security signals need somewhere to send an alert. Before the router, so
588 + // nothing the server answers is counted against an uninstalled sink.
589 + makenotwork::security_signals::install(state.db.clone(), state.email.clone());
590 +
587 591 // Build router (shared with integration tests via lib.rs)
588 592 let app = build_app(&state, session_layer)
593 + // Outside every per-route limiter, so a 429 the governor returned is
594 + // seen here. Counting only; the alert, if any, is spawned off the
595 + // request path.
596 + .layer(axum::middleware::from_fn(
597 + |req: Request<axum::body::Body>, next: axum::middleware::Next| async move {
598 + // Read the header rather than calling `extract_client_ip`: that
599 + // helper counts absences to warn about a missing Cloudflare
600 + // proxy, and a per-response call would double every count.
601 + let ip = req
602 + .headers()
603 + .get("cf-connecting-ip")
604 + .and_then(|v| v.to_str().ok())
605 + .and_then(|s| s.split(',').next())
606 + .map(|s| s.trim().to_string())
607 + .filter(|s| !s.is_empty());
608 + let response = next.run(req).await;
609 + makenotwork::security_signals::note_response(
610 + response.status().as_u16(),
611 + ip.as_deref(),
612 + );
613 + response
614 + },
615 + ))
589 616 // Request ID: propagate → trace → set (Axum applies inside-out)
590 617 .layer(PropagateRequestIdLayer::x_request_id())
591 618 .layer(
M server/src/db/admin_alerts.rs +8
@@ -31,6 +31,13 @@
31 31 Route,
32 32 Scan,
33 33 Monitoring,
34 + /// Security signals raised by this server about itself: error-rate spikes,
35 + /// auth-failure spikes, rate-limit volume, webhook signature failures, CSP
36 + /// violations. One domain rather than one variant per condition, following
37 + /// the rule above: the sub-condition rides in `dedup_key` and the urgency in
38 + /// [`AlertSeverity`], exactly the way `Tls` already folds three PoM
39 + /// categories. See [`crate::security_signals`].
40 + Security,
34 41 }
35 42
36 43 impl AlertKind {
@@ -48,6 +55,7 @@
48 55 Self::Route => "route",
49 56 Self::Scan => "scan",
50 57 Self::Monitoring => "monitoring",
58 + Self::Security => "security",
51 59 }
52 60 }
53 61 }
M server/src/routes/auth.rs +11
@@ -118,7 +118,13 @@
118 118 };
119 119
120 120 let sso_enabled = config.sso.is_some();
121 + // Every failed login leaves through here, whatever the reason, and none of
122 + // them is visible in the response status: a wrong password re-renders the
123 + // form with a 200. Counting at the exit is what makes a stuffing run
124 + // countable at all.
125 + let failure_ip = crate::helpers::extract_client_ip(&headers);
121 126 let return_error = |msg: &str| -> Result<Response> {
127 + crate::security_signals::note_auth_failure(failure_ip.as_deref());
122 128 if is_htmx {
123 129 Ok(Html(
124 130 LoginErrorTemplate {
@@ -536,6 +542,11 @@
536 542 let auth_result = webauthn
537 543 .finish_discoverable_authentication(&auth, auth_state, &[discoverable_key])
538 544 .map_err(|e| {
545 + // A rejected assertion is a failed auth attempt, and it leaves as a
546 + // 400, so nothing about the response says so either.
547 + crate::security_signals::note_auth_failure(
548 + crate::helpers::extract_client_ip(&headers).as_deref(),
549 + );
539 550 if matches!(e, WebauthnError::CredentialPossibleCompromise) {
540 551 tracing::warn!(
541 552 user_id = %user_id,
M server/src/routes/api/csp_report.rs +55
@@ -85,6 +85,25 @@
85 85 document = %document,
86 86 "CSP violation reported"
87 87 );
88 +
89 + // The log target above was the whole control until now, and a log nobody
90 + // reads is how the last violation went unnoticed on every page load. Give
91 + // it a consumer.
92 + //
93 + // Only an explicitly enforced violation alerts, and the strictness is the
94 + // point. The Report-Only policy shipped alongside the enforced one drops
95 + // `'unsafe-inline'` from `style-src` while the templates still use inline
96 + // `style=""`, so it fires on ordinary traffic by design; alerting on it
97 + // would page about known work in progress. The legacy `report-uri` shape
98 + // carries no `disposition` at all, so a Safari report cannot be told apart
99 + // from a Report-Only one and is treated as the latter. That loses
100 + // Safari-only enforced violations, which is the right way round: every
101 + // other browser sends `report-to` with a disposition, so a real enforced
102 + // violation still arrives, and the alternative is a daily alert about the
103 + // candidate policy.
104 + if disposition == "enforce" {
105 + crate::security_signals::note_csp_violation(&blocked, &directive, &document);
106 + }
88 107 }
89 108
90 109 fn truncate(value: &str) -> String {
@@ -191,6 +210,42 @@
191 210 );
192 211 }
193 212
213 + /// The Report-Only policy fires on ordinary traffic on purpose (it drops
214 + /// `'unsafe-inline'` from `style-src` while the templates still need it),
215 + /// so alerting on it would page daily about known work. Only `enforce`
216 + /// counts, and a report with no `disposition` is treated as Report-Only.
217 + #[test]
218 + fn only_enforced_violations_reach_the_alert_path() {
219 + let enforced = serde_json::json!({
220 + "documentURL": "https://makenot.work/",
221 + "effectiveDirective": "script-src",
222 + "blockedURL": "https://evil.example/x.js",
223 + "disposition": "enforce",
224 + });
225 + let report_only = serde_json::json!({
226 + "documentURL": "https://makenot.work/",
227 + "effectiveDirective": "style-src",
228 + "blockedURL": "inline",
229 + "disposition": "report",
230 + });
231 + // Legacy report-uri: no disposition field at all.
232 + let legacy = serde_json::json!({
233 + "document-uri": "https://makenot.work/",
234 + "violated-directive": "style-src",
235 + "blocked-uri": "inline",
236 + });
237 +
238 + let disposition = |v: &Value| -> String {
239 + v.get("disposition")
240 + .and_then(Value::as_str)
241 + .unwrap_or_default()
242 + .to_string()
243 + };
244 + assert_eq!(disposition(&enforced), "enforce");
245 + assert_ne!(disposition(&report_only), "enforce");
246 + assert_ne!(disposition(&legacy), "enforce");
247 + }
248 +
194 249 #[test]
195 250 fn truncates_long_fields_on_a_char_boundary() {
196 251 let long = "\u{e9}".repeat(MAX_FIELD_LEN);
M server/src/routes/api/mod.rs +1 -1
@@ -28,7 +28,7 @@
28 28 pub(crate) mod git_tokens;
29 29 mod guest_checkout;
30 30 mod imports;
31 - mod internal;
31 + pub(crate) mod internal;
32 32 mod items;
33 33 pub(crate) mod license_keys;
34 34 mod links;
M server/src/routes/stripe/webhook_v2.rs +8 -2
@@ -42,8 +42,14 @@
42 42 let payload = std::str::from_utf8(&body)
43 43 .map_err(|_| AppError::BadRequest("Invalid payload encoding".to_string()))?;
44 44
45 - // Verify signature and parse JSON
46 - let body_json = stripe.verify_webhook_v2(payload, signature)?;
45 + // Verify signature and parse JSON. A failure here has no benign cause:
46 + // Stripe signs correctly, so it means a wrong signing secret (real events
47 + // being dropped) or forged events aimed at the billing path.
48 + let body_json = stripe
49 + .verify_webhook_v2(payload, signature)
50 + .inspect_err(|_| {
51 + crate::security_signals::note_webhook_signature_failure("stripe");
52 + })?;
47 53
48 54 // Parse the thin event
49 55 let thin: ThinEvent = serde_json::from_value(body_json).map_err(|e| {
M server/src/routes/api/internal/alerts.rs +63 -29
@@ -40,6 +40,58 @@
40 40 details: Option<serde_json::Value>,
41 41 }
42 42
43 + /// Mail an already-persisted alert to the operator and mark the row emailed.
44 + ///
45 + /// Shared with [`crate::security_signals`], which raises alerts about this
46 + /// server from inside this process. That path calls `insert_alert` directly
47 + /// rather than posting to the handler above: the endpoint exists for external
48 + /// agents, and signalling ourselves over the loopback would be a network hop
49 + /// and a bearer-token round trip to reach a function we can call. Both paths
50 + /// have to produce the same row and the same mail, so the mail half lives here
51 + /// instead of being written twice.
52 + ///
53 + /// Never returns an error. The row is already durable, and a mail failure is
54 + /// logged rather than surfaced: to an ingesting agent a 5xx means retry, which
55 + /// would duplicate the row.
56 + #[allow(clippy::too_many_arguments)]
57 + pub(crate) async fn email_alert(
58 + pool: &PgPool,
59 + email: &EmailClient,
60 + id: i64,
61 + source: &str,
62 + kind: AlertKind,
63 + severity: AlertSeverity,
64 + title: &str,
65 + body: &str,
66 + ) {
67 + match std::env::var("ALERT_EMAIL").ok() {
68 + Some(to) if !to.is_empty() => {
69 + let subject = format!("[{}/{}] {}", source, severity.as_str(), title);
70 + let mail_body = format!(
71 + "{}\n\nsource: {}\nkind: {}\nseverity: {}\n",
72 + body,
73 + source,
74 + kind.as_str(),
75 + severity.as_str()
76 + );
77 + match email.send_alert(&to, &subject, &mail_body).await {
78 + Ok(()) => {
79 + if let Err(e) = db::admin_alerts::mark_emailed(pool, id).await {
80 + tracing::error!(alert_id = id, error = ?e, "alert emailed but mark_emailed failed");
81 + }
82 + }
83 + Err(e) => {
84 + tracing::error!(alert_id = id, error = ?e, "failed to email ingested alert");
85 + }
86 + }
87 + }
88 + _ => tracing::warn!(
89 + alert_id = id,
90 + "ALERT_EMAIL unset; alert logged but not emailed"
91 + ),
92 + }
93 + }
94 +
43 95 fn require_field(name: &str, value: &str, max: usize) -> Result<()> {
44 96 if value.is_empty() {
45 97 return Err(AppError::validation(format!("{name} is required")));
@@ -92,35 +144,17 @@
92 144 )
93 145 .await?;
94 146
95 - match std::env::var("ALERT_EMAIL").ok() {
96 - Some(to) if !to.is_empty() => {
97 - let subject = format!("[{}/{}] {}", req.source, req.severity.as_str(), req.title);
98 - let mail_body = format!(
99 - "{}\n\nsource: {}\nkind: {}\nseverity: {}\n",
100 - req.body,
101 - req.source,
102 - req.kind.as_str(),
103 - req.severity.as_str()
104 - );
105 - match email.send_alert(&to, &subject, &mail_body).await {
106 - Ok(()) => {
107 - if let Err(e) = db::admin_alerts::mark_emailed(&pool, id).await {
108 - tracing::error!(alert_id = id, error = ?e, "alert emailed but mark_emailed failed");
109 - }
110 - }
111 - // Ingestion succeeded and the row is durable; a mail failure is
112 - // logged, not surfaced as a 5xx to the agent (which would make it
113 - // retry and duplicate the row).
114 - Err(e) => {
115 - tracing::error!(alert_id = id, error = ?e, "failed to email ingested alert");
116 - }
117 - }
118 - }
119 - _ => tracing::warn!(
120 - alert_id = id,
121 - "ALERT_EMAIL unset; alert logged but not emailed"
122 - ),
123 - }
147 + email_alert(
148 + &pool,
149 + &email,
150 + id,
151 + &req.source,
152 + req.kind,
153 + req.severity,
154 + &req.title,
155 + &req.body,
156 + )
157 + .await;
124 158
125 159 Ok((
126 160 axum::http::StatusCode::ACCEPTED,
M server/src/routes/api/internal/mod.rs +1 -1
@@ -3,7 +3,7 @@
3 3 //! These endpoints are protected by `ServiceAuth` (Bearer token) and are
4 4 //! called by the CLI SSH server running on the same host.
5 5
6 - mod alerts;
6 + pub(crate) mod alerts;
7 7 mod cli_features;
8 8 mod content;
9 9 mod creators;
M server/src/routes/stripe/webhook/mod.rs +5 -1
@@ -54,7 +54,11 @@
54 54 let payload = std::str::from_utf8(&body)
55 55 .map_err(|_| AppError::BadRequest("Invalid payload encoding".to_string()))?;
56 56
57 - let event = stripe.verify_webhook(payload, signature)?;
57 + // A failure here has no benign cause: Stripe signs correctly, so it means
58 + // a wrong signing secret (real events being dropped) or forged events.
59 + let event = stripe.verify_webhook(payload, signature).inspect_err(|_| {
60 + crate::security_signals::note_webhook_signature_failure("stripe");
61 + })?;
58 62 tracing::info!(event_type = %event.type_, event_id = %event.id, "received webhook event");
59 63
60 64 // Serialize concurrent redeliveries of this event id. Held across the whole
A server/src/security_signals.rs +500
@@ -1,0 +1,637 @@
1 + //! Alerting on security signals, so the logs are not the only place they land.
2 + //!
3 + //! The 2026-07-21 audit collapsed sixty findings into one root cause: nothing
4 + //! tells you when something breaks. CSP reporting was that bug at page level and
5 + //! the last violation was found by a human opening a browser console. This is
6 + //! the same bug one layer up. A 5xx spike, a credential-stuffing run against
7 + //! login, a webhook arriving with a bad Stripe signature: each is logged, and a
8 + //! log nobody reads is not a control.
9 + //!
10 + //! Deliberately not a monitoring product. No metrics stack, no time series, no
11 + //! dashboard, no second thing to keep alive on the box. Counters live in this
12 + //! process, in fixed windows, and cross a threshold at most once per window.
13 + //! Delivery reuses what `POST /api/internal/alerts` already built: a row in
14 + //! `admin_alerts` and mail to `ALERT_EMAIL`. That endpoint exists for external
15 + //! agents (PoM, MT), so signals raised here call the same insert-and-mail path
16 + //! in process rather than posting to ourselves over the loopback with a bearer
17 + //! token. `ALERTS_INGEST_TOKEN` gates inbound requests from other machines and
18 + //! is not a precondition for anything in this file.
19 + //!
20 + //! Thresholds are numbers, and every one of them is reversible: noisy, raise
21 + //! it; quiet, lower it. Nothing here is structural.
22 + //!
23 + //! **Suppression is load-bearing.** Nothing dedups server-side --
24 + //! `insert_alert` is a plain INSERT and `admin_alerts` has no unique constraint
25 + //! on `dedup_key` (migration 169 says deduplication is the sending agent's job).
26 + //! The once-per-window rule below is therefore the only thing standing between
27 + //! one condition and a mailbox full of identical alerts, which is why
28 + //! `crossing_twice_in_one_window_alerts_once` is a real test and not a
29 + //! formality.
30 + //!
31 + //! <!-- wiki: mnw-server-overview -->
32 +
33 + use std::collections::{HashMap, HashSet};
34 + use std::sync::{Mutex, OnceLock};
35 + use std::time::{Duration, Instant};
36 +
37 + use crate::db::admin_alerts::{AlertKind, AlertSeverity, NewAlert};
38 + use crate::email::EmailClient;
39 +
40 + /// Counting window for everything except CSP. Long enough that a threshold
41 + /// means a rate, short enough that the operator hears about it while it is
42 + /// still happening.
43 + const WINDOW: Duration = Duration::from_mins(5);
44 +
45 + /// CSP violations are a page-level defect, not a rate: the same blocked URI
46 + /// fires on every load until someone fixes the page. Once a day per distinct
47 + /// URI says so without saying it four thousand times.
48 + const CSP_WINDOW: Duration = Duration::from_hours(24);
49 +
50 + /// Fraction of responses that must be 5xx before the error rate alerts.
51 + const ERROR_RATE_THRESHOLD: f64 = 0.01;
52 +
53 + /// Requests a window needs before its error rate is meaningful. Without this, a
54 + /// quiet night with two requests and one error reads as a 50% error rate.
55 + const ERROR_RATE_MIN_REQUESTS: u64 = 20;
56 +
57 + /// Failed auth attempts from one address before it looks like stuffing rather
58 + /// than a forgotten password.
59 + const AUTH_FAILURES_PER_IP: u64 = 20;
60 +
61 + /// Failed auth attempts across all addresses. Catches the distributed version,
62 + /// which the per-IP threshold is blind to by construction.
63 + const AUTH_FAILURES_SITE_WIDE: u64 = 100;
64 +
65 + /// Rate-limit trips before the volume is worth hearing about. An individual
66 + /// trip is the system working exactly as designed and must never alert.
67 + const RATE_LIMIT_TRIPS: u64 = 50;
68 +
69 + /// Everything needed to raise an alert, captured once at startup.
70 + ///
71 + /// A global rather than state threaded through every call site: the CSP report
72 + /// handler takes a request body and nothing else, and the response middleware
73 + /// runs outside the state extractors. Uninitialised (every unit test, and any
74 + /// build that never calls [`install`]) means counting still happens and nothing
75 + /// is sent, which is the correct degraded behaviour for a best-effort channel.
76 + struct Sink {
77 + db: sqlx::PgPool,
78 + email: EmailClient,
79 + }
80 +
81 + static SINK: OnceLock<Sink> = OnceLock::new();
82 +
83 + /// Point the signal counters at a database and a mailer. Call once at startup.
84 + /// A second call is ignored rather than treated as an error, so the per-test
85 + /// `build_app` does not have to care.
86 + pub fn install(db: sqlx::PgPool, email: EmailClient) {
87 + let _ = SINK.set(Sink { db, email });
88 + }
89 +
90 + /// A condition that has crossed its threshold and is worth one alert.
91 + #[derive(Debug, Clone)]
92 + struct Firing {
93 + severity: AlertSeverity,
94 + dedup_key: String,
95 + title: String,
96 + body: String,
97 + }
98 +
99 + /// Fixed-window counters. One mutex: every field is touched together on the
100 + /// response path and the critical section is a handful of integer adds, so
101 + /// splitting it would buy contention rather than remove it.
102 + #[derive(Default)]
103 + struct Counters {
104 + window_started: Option<Instant>,
105 + requests: u64,
106 + server_errors: u64,
107 + auth_failures: u64,
108 + auth_failures_by_ip: HashMap<String, u64>,
109 + rate_limit_trips: u64,
110 + /// Conditions already alerted on in this window. Cleared with the window.
111 + fired: HashSet<String>,
112 +
113 + csp_window_started: Option<Instant>,
114 + /// Blocked URIs already alerted on today. Bounded by [`CSP_URI_CAP`] so a
115 + /// page generating unique blocked URIs cannot grow this without limit.
116 + csp_fired: HashSet<String>,
117 + }
118 +
119 + /// Distinct blocked URIs tracked per CSP window. A violation carrying a
120 + /// cache-busted or otherwise unique URI each time would otherwise make this set
121 + /// a slow memory leak; past the cap, further distinct URIs are counted as
122 + /// already-alerted rather than remembered.
123 + const CSP_URI_CAP: usize = 256;
124 +
125 + static COUNTERS: Mutex<Option<Counters>> = Mutex::new(None);
126 +
127 + /// Run `f` against the counters, rolling the window first if it has expired.
128 + ///
129 + /// A poisoned mutex is not propagated. Every caller is on a request path that
130 + /// has real work to do, and failing a checkout because an alert counter panicked
131 + /// in another thread would make this module the outage it exists to report.
132 + fn with_counters<T>(now: Instant, f: impl FnOnce(&mut Counters) -> T) -> Option<T> {
133 + let mut guard = COUNTERS.lock().ok()?;
134 + let counters = guard.get_or_insert_with(Counters::default);
135 +
136 + match counters.window_started {
137 + Some(started) if now.duration_since(started) < WINDOW => {}
138 + _ => {
139 + counters.window_started = Some(now);
140 + counters.requests = 0;
141 + counters.server_errors = 0;
142 + counters.auth_failures = 0;
143 + counters.auth_failures_by_ip.clear();
144 + counters.rate_limit_trips = 0;
145 + counters.fired.clear();
146 + }
147 + }
148 +
149 + match counters.csp_window_started {
150 + Some(started) if now.duration_since(started) < CSP_WINDOW => {}
151 + _ => {
152 + counters.csp_window_started = Some(now);
153 + counters.csp_fired.clear();
154 + }
155 + }
156 +
157 + Some(f(counters))
158 + }
159 +
160 + impl Counters {
161 + /// Record that `key` has fired, returning false if it already had this
162 + /// window. The whole suppression rule, in one place.
163 + fn claim(&mut self, key: &str) -> bool {
164 + self.fired.insert(key.to_string())
165 + }
166 + }
167 +
168 + /// Observe one finished response.
169 + ///
170 + /// Called from the outermost middleware, so it sees every request the server
171 + /// answered, including the ones rejected by a layer before any handler ran.
172 + /// `ip` is the Cloudflare-derived client address where one is available.
173 + pub fn note_response(status: u16, ip: Option<&str>) {
174 + note_response_at(status, ip, Instant::now());
175 + }
176 +
177 + fn note_response_at(status: u16, ip: Option<&str>, now: Instant) {
178 + let firing = with_counters(now, |c| {
179 + c.requests += 1;
180 +
181 + if status >= 500 {
182 + c.server_errors += 1;
183 + let rate = c.server_errors as f64 / c.requests as f64;
184 + if c.requests >= ERROR_RATE_MIN_REQUESTS
185 + && rate > ERROR_RATE_THRESHOLD
186 + && c.claim("sec:5xx")
187 + {
188 + return Some(Firing {
189 + severity: AlertSeverity::Critical,
190 + dedup_key: "sec:5xx".to_string(),
191 + title: "Server error rate is elevated".to_string(),
192 + body: format!(
193 + "{} of {} responses in the last {} minutes were 5xx ({:.1}%). \
194 + Threshold is {:.0}% over at least {} requests.",
195 + c.server_errors,
196 + c.requests,
197 + WINDOW.as_secs() / 60,
198 + rate * 100.0,
199 + ERROR_RATE_THRESHOLD * 100.0,
200 + ERROR_RATE_MIN_REQUESTS,
201 + ),
202 + });
203 + }
204 + }
205 +
206 + if status == 429 {
207 + c.rate_limit_trips += 1;
208 + if c.rate_limit_trips > RATE_LIMIT_TRIPS && c.claim("sec:ratelimit") {
209 + return Some(Firing {
210 + severity: AlertSeverity::Warning,
211 + dedup_key: "sec:ratelimit".to_string(),
212 + title: "Rate limiting is tripping in volume".to_string(),
213 + body: format!(
214 + "{} requests were rate limited in the last {} minutes (threshold {}). \
215 + Individual trips are the limiter working; this many suggests a scraper, \
216 + a stuck client, or a limit set too tight.",
217 + c.rate_limit_trips,
218 + WINDOW.as_secs() / 60,
219 + RATE_LIMIT_TRIPS,
220 + ),
221 + });
222 + }
223 + }
224 +
225 + let _ = ip;
226 + None
227 + })
228 + .flatten();
229 +
230 + dispatch(firing);
231 + }
232 +
233 + /// Record one failed authentication attempt.
234 + ///
235 + /// Called at the failure sites rather than inferred from status codes: a wrong
236 + /// password re-renders the login form with a 200, so nothing about the response
237 + /// says an attempt failed.
238 + pub fn note_auth_failure(ip: Option<&str>) {
239 + note_auth_failure_at(ip, Instant::now());
240 + }
241 +
242 + fn note_auth_failure_at(ip: Option<&str>, now: Instant) {
243 + let firing = with_counters(now, |c| {
244 + c.auth_failures += 1;
245 +
246 + if let Some(ip) = ip {
247 + // One entry per address per window, and the window clears it. A
248 + // spray from many addresses is bounded by the same window rather
249 + // than by a cap, which is what the site-wide counter is for.
250 + let per_ip = c.auth_failures_by_ip.entry(ip.to_string()).or_insert(0);
251 + *per_ip += 1;
252 + let count = *per_ip;
253 + let key = format!("sec:authfail:{ip}");
254 + if count > AUTH_FAILURES_PER_IP && c.claim(&key) {
255 + return Some(Firing {
256 + severity: AlertSeverity::Critical,
257 + dedup_key: key,
258 + title: format!("Repeated auth failures from {ip}"),
259 + body: format!(
260 + "{count} failed authentication attempts from {ip} in the last {} minutes \
261 + (threshold {AUTH_FAILURES_PER_IP}). Looks like credential stuffing rather \
262 + than a forgotten password.",
263 + WINDOW.as_secs() / 60,
264 + ),
265 + });
266 + }
267 + }
268 +
269 + if c.auth_failures > AUTH_FAILURES_SITE_WIDE && c.claim("sec:authfail") {
270 + return Some(Firing {
271 + severity: AlertSeverity::Critical,
272 + dedup_key: "sec:authfail".to_string(),
273 + title: "Auth failures are elevated site-wide".to_string(),
274 + body: format!(
275 + "{} failed authentication attempts across all addresses in the last {} minutes \
276 + (threshold {AUTH_FAILURES_SITE_WIDE}). A distributed attempt would look like \
277 + this and would not trip the per-address threshold.",
278 + c.auth_failures,
279 + WINDOW.as_secs() / 60,
280 + ),
281 + });
282 + }
283 +
284 + None
285 + })
286 + .flatten();
287 +
288 + dispatch(firing);
289 + }
290 +
291 + /// Record a webhook that arrived with a signature that did not verify.
292 + ///
293 + /// No threshold: there is no benign cause. Stripe is the caller and it signs
294 + /// correctly, so one of these means either a misconfigured secret or someone
295 + /// posting forged events at the billing path.
296 + pub fn note_webhook_signature_failure(provider: &str) {
297 + note_webhook_signature_failure_at(provider, Instant::now());
298 + }
299 +
300 + fn note_webhook_signature_failure_at(provider: &str, now: Instant) {
301 + let provider = provider.to_string();
302 + let firing = with_counters(now, |c| {
303 + let key = format!("sec:webhook:{provider}");
304 + if !c.claim(&key) {
305 + return None;
306 + }
307 + Some(Firing {
308 + severity: AlertSeverity::Critical,
309 + dedup_key: key,
310 + title: format!("{provider} webhook signature failed to verify"),
311 + body: format!(
312 + "A request to the {provider} webhook path carried a signature that did not \
313 + verify. There is no benign cause: either the signing secret is wrong (in which \
314 + case real events are being dropped) or someone is posting forged events."
315 + ),
316 + })
317 + })
318 + .flatten();
319 +
320 + dispatch(firing);
321 + }
322 +
323 + /// Record a CSP violation report, alerting once per distinct blocked URI per
324 + /// day. Gives the reporting added on 2026-07-28 a consumer.
325 + pub fn note_csp_violation(blocked_uri: &str, directive: &str, document: &str) {
326 + note_csp_violation_at(blocked_uri, directive, document, Instant::now());
327 + }
328 +
329 + fn note_csp_violation_at(blocked_uri: &str, directive: &str, document: &str, now: Instant) {
330 + let firing = with_counters(now, |c| {
331 + if c.csp_fired.len() >= CSP_URI_CAP || !c.csp_fired.insert(blocked_uri.to_string()) {
332 + return None;
333 + }
334 + Some(Firing {
335 + severity: AlertSeverity::Warning,
336 + dedup_key: format!("sec:csp:{blocked_uri}"),
337 + title: format!("CSP violation: {directive}"),
338 + body: format!(
339 + "A page reported a Content-Security-Policy violation.\n\n\
340 + directive: {directive}\nblocked: {blocked_uri}\ndocument: {document}\n\n\
341 + Either something on the page broke, or someone is probing. Reported once per \
342 + blocked URI per day."
343 + ),
344 + })
345 + })
346 + .flatten();
347 +
348 + dispatch(firing);
349 + }
350 +
351 + /// Persist and mail a firing, off the request path.
352 + ///
353 + /// Best effort in the strict sense: a failure here logs and goes no further. It
354 + /// never blocks the request, never returns an error to a caller, and never
355 + /// panics. An alerting channel that can take the site down is worse than no
356 + /// alerting channel.
357 + fn dispatch(firing: Option<Firing>) {
358 + let Some(firing) = firing else { return };
359 + let Some(sink) = SINK.get() else {
360 + // No sink: unit tests and any build that skipped `install`. The
361 + // threshold logic still ran, which is what those tests assert.
362 + tracing::debug!(
363 + dedup_key = %firing.dedup_key,
364 + "security signal fired before the alert sink was installed"
365 + );
366 + return;
367 + };
368 +
369 + let db = sink.db.clone();
370 + let email = sink.email.clone();
371 + tokio::spawn(async move {
372 + tracing::warn!(
373 + target: "security_signal",
374 + dedup_key = %firing.dedup_key,
375 + severity = firing.severity.as_str(),
376 + "{}",
377 + firing.title
378 + );
379 +
380 + let id = match crate::db::admin_alerts::insert_alert(
381 + &db,
382 + &NewAlert {
383 + source: "mnw",
384 + kind: AlertKind::Security,
385 + severity: firing.severity,
386 + title: &firing.title,
387 + body: &firing.body,
388 + dedup_key: Some(&firing.dedup_key),
389 + details: None,
390 + },
391 + )
392 + .await
393 + {
394 + Ok(id) => id,
395 + Err(e) => {
396 + tracing::error!(error = ?e, "failed to persist security alert");
397 + return;
398 + }
399 + };
400 +
401 + crate::routes::api::internal::alerts::email_alert(
402 + &db,
403 + &email,
404 + id,
405 + "mnw",
406 + AlertKind::Security,
407 + firing.severity,
408 + &firing.title,
409 + &firing.body,
410 + )
411 + .await;
412 + });
413 + }
414 +
415 + #[cfg(test)]
416 + mod tests {
417 + use super::*;
418 +
419 + /// Every test drives the counters through the `_at` variants with an
420 + /// explicit clock, and they share one global. Serialise them rather than
421 + /// letting a stray count from a parallel test move a threshold.
422 + fn lock() -> std::sync::MutexGuard<'static, ()> {
423 + static SERIAL: Mutex<()> = Mutex::new(());
424 + SERIAL
425 + .lock()
426 + .unwrap_or_else(std::sync::PoisonError::into_inner)
427 + }
428 +
429 + /// Force a fresh window. The counters are global, so a test that assumes an
430 + /// empty window has to say so.
431 + fn reset() {
432 + if let Ok(mut guard) = COUNTERS.lock() {
433 + *guard = None;
434 + }
435 + }
436 +
437 + /// What `fired` holds, which is the observable form of "an alert was sent"
438 + /// without a database behind it.
439 + fn fired() -> HashSet<String> {
440 + COUNTERS
441 + .lock()
442 + .unwrap()
443 + .as_ref()
444 + .map(|c| c.fired.clone())
445 + .unwrap_or_default()
446 + }
447 +
448 + #[test]
449 + fn error_rate_needs_volume_before_it_alerts() {
450 + let _g = lock();
451 + reset();
452 + let t = Instant::now();
453 +
454 + // Two requests, one of them a 500. That is a 50% error rate and it must
455 + // not alert: below the minimum request count, the ratio is noise.
456 + note_response_at(500, None, t);
457 + note_response_at(200, None, t);
458 + assert!(!fired().contains("sec:5xx"), "alerted on two requests");
459 + }
460 +
461 + #[test]
462 + fn error_rate_alerts_once_past_the_threshold() {
463 + let _g = lock();
464 + reset();
465 + let t = Instant::now();
466 +
467 + for _ in 0..ERROR_RATE_MIN_REQUESTS {
468 + note_response_at(200, None, t);
469 + }
470 + assert!(!fired().contains("sec:5xx"), "clean traffic alerted");
471 +
472 + note_response_at(500, None, t);
473 + assert!(fired().contains("sec:5xx"), "1 in 21 is over 1%");
474 + }
475 +
476 + #[test]
477 + fn crossing_twice_in_one_window_alerts_once() {
478 + let _g = lock();
479 + reset();
480 + let t = Instant::now();
481 +
482 + for _ in 0..ERROR_RATE_MIN_REQUESTS {
483 + note_response_at(200, None, t);
484 + }
485 + note_response_at(500, None, t);
486 + assert!(fired().contains("sec:5xx"));
487 +
488 + // Nothing dedups server-side, so this is the only suppression there is.
489 + // Staying over the threshold must not produce a second alert.
490 + let before = fired().len();
491 + for _ in 0..50 {
492 + note_response_at(500, None, t);
493 + }
494 + assert_eq!(fired().len(), before, "a sustained condition realerted");
495 + }
496 +
497 + #[test]
498 + fn the_window_resets_and_can_alert_again() {
499 + let _g = lock();
500 + reset();
Lines truncated