max / makenotwork

pom: watch the host CA bundle, and stop discarding the platform trust result The TLS check has always probed each target against two trust stores, the bundled web PKI and the host's own, then folded the first into `valid` and dropped the second. Nothing read `platform_trusted`: not the ledger, not the status payload, not an alert. A host trust store that rejects a chain the public web PKI accepts is exactly the failure multithreaded hits when the CA bundle goes stale or thin, since it takes its outbound anchors from that bundle on all three of its outbound paths with no in-binary fallback, and the reading that would have caught it was being computed every five minutes and thrown away. Persist all four trust fields (migration 14, nullable so pre-migration rows say "no reading" rather than "untrusted"), and make the divergence a degraded condition on the target. Add checks::ca_bundle as the leading indicator, since by the time a handshake fails the outage has started. Three signals: installed versus candidate package version, age of the last successful package-list update, and the certificate count against a floor. The second is the one worth being explicit about: without it, installed == candidate stays true forever the moment apt-daily.timer stops running, and the check reports a green it has no evidence for. The floor catches a truncated bundle that both version numbers call fine. Severity split follows what the operator can do: thin is red (outbound TLS is broken now), stale and unknown are yellow (drift). Alerts fold onto the "tls" MNW domain rather than adding an AlertKind the ingest would 422. Verified against the live host: on a box with no apt periodic stamps the probe reports unknown rather than a false ok. Refs infra eb4abe8f.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-06 19:39 UTC

Signed with PGP, not checked

Commit: 49c87664950aec81614f89009773b91f3192536c

Parent: d0f7f48

20 files changed, +1099 insertions, -33 deletions

M pom/pom.toml +13

			@@ -9,3 +9,16 @@
9	9		[targets.mnw.health]
10	10		url = "https://makenot.work/api/health"
11	11		timeout_secs = 10
	12	+
	13	+	# Host trust-anchor freshness. Local to whichever machine PoM runs on, like
	14	+	# [targets..systemd] and [targets..backups], so it belongs on the instance
	15	+	# that runs the workloads rather than on a monitoring box watching from outside.
	16	+	# Every field has a working default; the section only has to exist.
	17	+	#
	18	+	# [targets.mnw.ca_bundle]
	19	+	# package = "ca-certificates"
	20	+	# bundle_path = "/etc/ssl/certs/ca-certificates.crt"
	21	+	# min_certs = 80
	22	+	# update_stamp = "/var/lib/apt/periodic/update-success-stamp"
	23	+	# update_stamp_max_age_hours = 48
	24	+	# interval_secs = 3600

M pom/docs/architecture.md +5 -3

			@@ -17,12 +17,12 @@
17	17		\| Module \| File \| Role \|
18	18		\|--------\|------\|------\|
19	19		\| `main` \| `src/main.rs` \| Entry point: parses CLI args, dispatches to CLI handler or MCP server \|
20		-	\| `cli` \| `src/cli/` \| CLI command handlers. `mod.rs` dispatch, plus `serve.rs`, `status.rs`, `incident.rs`, `transition.rs`, and `tasks/` (one spawner per background loop: health, tls, dns, whois, routes, cors, backup, systemd, scan_pipeline, synckit_fleet, meta_alert, prune) \|
	20	+	\| `cli` \| `src/cli/` \| CLI command handlers. `mod.rs` dispatch, plus `serve.rs`, `status.rs`, `incident.rs`, `transition.rs`, and `tasks/` (one spawner per background loop: health, tls, dns, whois, routes, cors, backup, systemd, ca_bundle, scan_pipeline, synckit_fleet, meta_alert, prune) \|
21	21		\| `config` \| `src/config.rs` \| TOML config loading, types for targets/peers/alerts/serve settings \|
22	22		\| `types` \| `src/types.rs` \| Shared domain types: HealthSnapshot, TestRun, TlsStatus, LatencyStats, TestStaleness \|
23		-	\| `db` \| `src/db/` \| SQLite persistence. `migrations.rs` holds the numbered schema migrations and pool construction; the rest is one query module per subject: health, test_runs, alerts, incidents, peers, tls, dns, whois, routes, cors, backup, systemd, scan_pipeline, synckit_fleet, maintenance \|
	23	+	\| `db` \| `src/db/` \| SQLite persistence. `migrations.rs` holds the numbered schema migrations and pool construction; the rest is one query module per subject: health, test_runs, alerts, incidents, peers, tls, dns, whois, routes, cors, backup, systemd, ca_bundle, scan_pipeline, synckit_fleet, maintenance \|
24	24		\| `api` \| `src/api.rs` \| Axum HTTP API: status, trends, peer info, mesh view, bearer token auth middleware \|
25		-	\| `alerts` \| `src/alerts/` \| Alerter struct: sends emails via Postmark on status transitions, with cooldown tracking. One module per alert kind: health, tls, latency, offline, peer, dns, whois, route, cors, backup, systemd, scan, test_duration \|
	25	+	\| `alerts` \| `src/alerts/` \| Alerter struct: sends emails via Postmark on status transitions, with cooldown tracking. One module per alert kind: health, tls, latency, offline, peer, dns, whois, route, cors, backup, systemd, ca_bundle, scan, test_duration \|
26	26		\| `peer` \| `src/peer.rs` \| Peer mesh: identity management, heartbeat loops, grace period state machine, mesh state \|
27	27		\| `status` \| `src/status.rs` \| PoM's projection onto the shared operator status payload \|
28	28		\| `dashboard` \| `src/dashboard.rs` \| Optional HTML dashboard served at `GET /` \|
			@@ -42,6 +42,7 @@
42	42		\| `checks::cors` \| `src/checks/cors.rs` \| CORS preflight: sends OPTIONS and checks Access-Control headers \|
43	43		\| `checks::backup` \| `src/checks/backup.rs` \| Local filesystem backup verification: scans for PostgreSQL backup files, checks recency \|
44	44		\| `checks::systemd` \| `src/checks/systemd.rs` \| Local systemd unit health: liveness, crash-loops, optionally any failed unit on the host \|
	45	+	\| `checks::ca_bundle` \| `src/checks/ca_bundle.rs` \| Local CA-bundle freshness: installed vs candidate package version, age of the last package-list update, certificate count against a floor \|
45	46		\| `checks::scan_pipeline` \| `src/checks/scan_pipeline.rs` \| Polls a makenotwork instance's upload scan health and applies the audit thresholds \|
46	47		\| `checks::synckit_fleet` \| `src/checks/synckit_fleet.rs` \| Polls a makenotwork instance for which SyncKit SDK versions are syncing \|
47	48		\| `tools` \| `src/tools/mod.rs` \| MCP server definition (PomServer), tool registration via rmcp \|
			@@ -137,6 +138,7 @@
137	138		\| `pending_alerts` \| Alert outbox with retry state \| alert_key, category, channel, subject, body, priority, source, attempts, next_retry_at \|
138	139		\| `scan_pipeline_checks` \| Upload scan queue depth and stuck-job counts \| target, status, issues, queue_pending, queue_running, queue_stuck, held_total, checked_at \|
139	140		\| `systemd_checks` \| systemd unit state on a host \| target, status, units, failed_units, issues, checked_at, error \|
	141	+	\| `ca_bundle_checks` \| Host trust-anchor freshness \| target, status, package, installed, candidate, cert_count, lists_age_hours, issues, checked_at, error \|
140	142		\| `synckit_fleet_checks` \| SyncKit device and version spread over a window \| target, window_days, devices, versions, checked_at, error \|
141	143
142	144		Pre-migration databases are detected by the presence of the `health_checks` table and stamped as v1 without re-running the initial migration.

M pom/src/api.rs +15

			@@ -583,6 +583,9 @@
583	583		days_remaining: r.days_remaining,
584	584		checked_at: r.checked_at,
585	585		error: r.error,
	586	+	webpki_trusted: r.webpki_trusted,
	587	+	platform_trusted: r.platform_trusted,
	588	+	platform_error: r.platform_error,
586	589		});
587	590
588	591		let incident = db::get_open_incident(pool, &name)
			@@ -642,6 +645,17 @@
642	645		error: s.error,
643	646		});
644	647
	648	+	let ca_bundle = db::get_latest_ca_bundle_check(pool, &name)
	649	+	.await
	650	+	.ok()
	651	+	.flatten()
	652	+	.map(\|c\| crate::status::CaBundleView {
	653	+	issues: c.issue_list(),
	654	+	status: c.status,
	655	+	checked_at: c.checked_at,
	656	+	error: c.error,
	657	+	});
	658	+
645	659		let synckit_fleet = db::get_latest_synckit_fleet_check(pool, &name)
646	660		.await
647	661		.ok()
			@@ -742,6 +756,7 @@
742	756		backups,
743	757		scan_pipeline,
744	758		systemd,
	759	+	ca_bundle,
745	760		synckit_fleet,
746	761		tests,
747	762		dns,

M pom/src/config.rs +67

			@@ -279,6 +279,73 @@
279	279		/// Local systemd daemon liveness / crash-loop / failed-unit check. `None`
280	280		/// disables. Probes the host PoM runs on, not a remote target.
281	281		pub systemd: Option<SystemdConfig>,
	282	+	/// Local CA-bundle freshness check. `None` disables. Like `systemd`, this
	283	+	/// probes the host PoM runs on rather than a remote target.
	284	+	pub ca_bundle: Option<CaBundleConfig>,
	285	+	}
	286	+
	287	+	/// Local CA-bundle freshness monitoring for a host target.
	288	+	///
	289	+	/// The trailing indicator for this already exists: the TLS check validates every
	290	+	/// target against the host trust store as well as the web PKI, so a bundle that
	291	+	/// has gone thin shows up as a chain the host rejects. This is the leading one.
	292	+	/// It answers "is the bundle drifting" before anything fails, which matters
	293	+	/// because multithreaded has no in-binary root fallback on any outbound path and
	294	+	/// would otherwise report the problem as a login outage.
	295	+	#[derive(Debug, Clone, Deserialize)]
	296	+	pub struct CaBundleConfig {
	297	+	/// Package providing the trust anchors. Defaults to "ca-certificates".
	298	+	#[serde(default = "default_ca_bundle_package")]
	299	+	pub package: String,
	300	+	/// Concatenated PEM bundle to count certificates in. Defaults to
	301	+	/// "/etc/ssl/certs/ca-certificates.crt".
	302	+	#[serde(default = "default_ca_bundle_path")]
	303	+	pub bundle_path: PathBuf,
	304	+	/// Fewer certificates than this reads as a truncated or emptied bundle.
	305	+	/// Defaults to 80, comfortably under the ~120 a stock Ubuntu carries and
	306	+	/// well above anything that would still be a working trust store.
	307	+	#[serde(default = "default_ca_bundle_min_certs")]
	308	+	pub min_certs: usize,
	309	+	/// Stamp file whose mtime records the last successful `apt update`.
	310	+	/// Defaults to "/var/lib/apt/periodic/update-success-stamp".
	311	+	#[serde(default = "default_ca_bundle_stamp")]
	312	+	pub update_stamp: PathBuf,
	313	+	/// A stamp older than this means the package lists are stale, so
	314	+	/// installed-equals-candidate proves nothing. Defaults to 48 hours.
	315	+	#[serde(default = "default_ca_bundle_stamp_max_age_hours")]
	316	+	pub update_stamp_max_age_hours: i64,
	317	+	/// Seconds between checks. Defaults to 1 hour.
	318	+	#[serde(default = "default_ca_bundle_interval")]
	319	+	pub interval_secs: u64,
	320	+	}
	321	+
	322	+	fn default_ca_bundle_package() -> String {
	323	+	"ca-certificates".to_string()
	324	+	}
	325	+
	326	+	fn default_ca_bundle_path() -> PathBuf {
	327	+	PathBuf::from("/etc/ssl/certs/ca-certificates.crt")
	328	+	}
	329	+
	330	+	fn default_ca_bundle_min_certs() -> usize {
	331	+	// Ubuntu 24.04 ships ~121. The floor is not a freshness signal, it catches a
	332	+	// truncated or half-written bundle, which a version comparison calls fine.
	333	+	80
	334	+	}
	335	+
	336	+	fn default_ca_bundle_stamp() -> PathBuf {
	337	+	PathBuf::from("/var/lib/apt/periodic/update-success-stamp")
	338	+	}
	339	+
	340	+	fn default_ca_bundle_stamp_max_age_hours() -> i64 {
	341	+	// apt-daily.timer runs daily, so 48h allows one missed run before saying so.
	342	+	48
	343	+	}
	344	+
	345	+	fn default_ca_bundle_interval() -> u64 {
	346	+	// 1 hour. The package moves a few times a year; this is about noticing
	347	+	// within a day, not within a minute.
	348	+	3600
282	349		}
283	350
284	351		/// A local git checkout to measure a target's live build against.

M pom/src/display.rs +18 -3

			@@ -345,7 +345,7 @@
345	345		/// Format prune results for CLI display.
346	346		pub fn format_prune(result: &PruneResult, days: i64) -> String {
347	347		format!(
348		-	"Pruned {} health checks, {} test runs, {} test details, {} peer heartbeats, {} alerts, {} TLS checks, {} incidents, {} route checks, {} DNS checks, {} WHOIS checks, {} backup checks, {} systemd checks, {} synckit fleet checks older than {} days.\n",
	348	+	"Pruned {} health checks, {} test runs, {} test details, {} peer heartbeats, {} alerts, {} TLS checks, {} incidents, {} route checks, {} DNS checks, {} WHOIS checks, {} backup checks, {} systemd checks, {} CA bundle checks, {} synckit fleet checks older than {} days.\n",
349	349		result.health,
350	350		result.tests,
351	351		result.test_details,
			@@ -358,6 +358,7 @@
358	358		result.whois,
359	359		result.backups,
360	360		result.systemd,
	361	+	result.ca_bundle,
361	362		result.synckit_fleet,
362	363		days
363	364		)
			@@ -924,6 +925,10 @@
924	925		issuer: "CN=Let's Encrypt".to_string(),
925	926		checked_at: "2026-03-11T00:00:00Z".to_string(),
926	927		error: None,
	928	+	webpki_trusted: Some(true),
	929	+	platform_trusted: Some(true),
	930	+	webpki_error: None,
	931	+	platform_error: None,
927	932		};
928	933		let out = format_status_target(
929	934		"mnw",
			@@ -957,6 +962,10 @@
957	962		issuer: "CN=Let's Encrypt".to_string(),
958	963		checked_at: "2026-03-11T00:00:00Z".to_string(),
959	964		error: None,
	965	+	webpki_trusted: Some(true),
	966	+	platform_trusted: Some(true),
	967	+	webpki_error: None,
	968	+	platform_error: None,
960	969		};
961	970		let out = format_status_target(
962	971		"mnw",
			@@ -989,6 +998,10 @@
989	998		issuer: String::new(),
990	999		checked_at: "2026-03-11T00:00:00Z".to_string(),
991	1000		error: Some("connection refused".to_string()),
	1001	+	webpki_trusted: Some(false),
	1002	+	platform_trusted: Some(false),
	1003	+	webpki_error: Some("TCP connect failed".to_string()),
	1004	+	platform_error: Some("TCP connect failed".to_string()),
992	1005		};
993	1006		let out = format_status_target(
994	1007		"mnw",
			@@ -1231,12 +1244,13 @@
1231	1244		whois: 2,
1232	1245		backups: 1,
1233	1246		systemd: 6,
	1247	+	ca_bundle: 2,
1234	1248		synckit_fleet: 7,
1235	1249		};
1236	1250		let out = format_prune(&result, 30);
1237	1251		assert_eq!(
1238	1252		out,
1239		-	"Pruned 5 health checks, 3 test runs, 15 test details, 10 peer heartbeats, 2 alerts, 1 TLS checks, 4 incidents, 0 route checks, 8 DNS checks, 2 WHOIS checks, 1 backup checks, 6 systemd checks, 7 synckit fleet checks older than 30 days.\n"
	1253	+	"Pruned 5 health checks, 3 test runs, 15 test details, 10 peer heartbeats, 2 alerts, 1 TLS checks, 4 incidents, 0 route checks, 8 DNS checks, 2 WHOIS checks, 1 backup checks, 6 systemd checks, 2 CA bundle checks, 7 synckit fleet checks older than 30 days.\n"
1240	1254		);
1241	1255		}
1242	1256
			@@ -1255,10 +1269,11 @@
1255	1269		whois: 0,
1256	1270		backups: 0,
1257	1271		systemd: 0,
	1272	+	ca_bundle: 0,
1258	1273		synckit_fleet: 0,
1259	1274		};
1260	1275		let out = format_prune(&result, 7);
1261		-	assert!(out.contains("Pruned 0 health checks, 0 test runs, 0 test details, 0 peer heartbeats, 0 alerts, 0 TLS checks, 0 incidents, 0 route checks, 0 DNS checks, 0 WHOIS checks, 0 backup checks, 0 systemd checks, 0 synckit fleet checks older than 7 days."));
	1276	+	assert!(out.contains("Pruned 0 health checks, 0 test runs, 0 test details, 0 peer heartbeats, 0 alerts, 0 TLS checks, 0 incidents, 0 route checks, 0 DNS checks, 0 WHOIS checks, 0 backup checks, 0 systemd checks, 0 CA bundle checks, 0 synckit fleet checks older than 7 days."));
1262	1277		}
1263	1278
1264	1279		// format_mesh

M pom/src/status.rs +196

			@@ -90,6 +90,8 @@
90	90		pub scan_pipeline: Option<ScanView>,
91	91		/// Latest local systemd daemon-health check. `None` if not monitored.
92	92		pub systemd: Option<SystemdView>,
	93	+	/// Latest local CA-bundle freshness check. `None` if not monitored.
	94	+	pub ca_bundle: Option<CaBundleView>,
93	95		/// Latest SyncKit field-version readout. `None` if not monitored.
94	96		pub synckit_fleet: Option<SyncKitFleetView>,
95	97		/// Latest test run and PoM's staleness verdict. `None` if the target has no
			@@ -114,6 +116,11 @@
114	116		pub days_remaining: i64,
115	117		pub checked_at: String,
116	118		pub error: Option<String>,
	119	+	/// Per-trust-store results. `None` means the reading predates migration 14,
	120	+	/// not that the probe failed. See [`tls_condition`].
	121	+	pub webpki_trusted: Option<bool>,
	122	+	pub platform_trusted: Option<bool>,
	123	+	pub platform_error: Option<String>,
117	124		}
118	125
119	126		pub(crate) struct IncidentView {
			@@ -156,6 +163,15 @@
156	163		pub error: Option<String>,
157	164		}
158	165
	166	+	pub(crate) struct CaBundleView {
	167	+	/// One of "ok", "stale", "thin", "unknown", "error".
	168	+	pub status: String,
	169	+	/// The why behind a non-ok status, one line per fired signal.
	170	+	pub issues: Vec<String>,
	171	+	pub checked_at: String,
	172	+	pub error: Option<String>,
	173	+	}
	174	+
159	175		/// A SyncKit field-version readout. No status field: the reading cannot be
160	176		/// unhealthy, only unavailable. See [`crate::checks::synckit_fleet`].
161	177		pub(crate) struct SyncKitFleetView {
			@@ -253,6 +269,9 @@
253	269		if let Some(sd) = &target.systemd {
254	270		conditions.push(systemd_condition(sd));
255	271		}
	272	+	if let Some(ca) = &target.ca_bundle {
	273	+	conditions.push(ca_bundle_condition(ca));
	274	+	}
256	275		if let Some(fleet) = &target.synckit_fleet {
257	276		conditions.push(synckit_fleet_condition(fleet));
258	277		}
			@@ -354,6 +373,24 @@
354	373		Status::Failed,
355	374		format!("certificate expired {} days ago", -tls.days_remaining),
356	375		)
	376	+	} else if tls.webpki_trusted == Some(true) && tls.platform_trusted == Some(false) {
	377	+	// The chain is fine by the public web PKI and rejected by this host's
	378	+	// own trust store, so the fault is the host's trust anchors rather than
	379	+	// the certificate. That is the CA-bundle failure multithreaded would hit
	380	+	// on all three of its outbound paths, and it used to arrive as a login
	381	+	// outage because this reading was computed and thrown away.
	382	+	//
	383	+	// Ordered ahead of the expiry warning deliberately: both are degraded,
	384	+	// and a broken trust store is the rarer and more actionable of the two.
	385	+	(
	386	+	Status::Degraded,
	387	+	match &tls.platform_error {
	388	+	Some(error) => {
	389	+	format!("host trust store rejects a chain the public web PKI accepts: {error}")
	390	+	}
	391	+	None => "host trust store rejects a chain the public web PKI accepts".into(),
	392	+	},
	393	+	)
357	394		} else if tls.days_remaining <= TLS_EXPIRY_WARN_DAYS {
358	395		(
359	396		Status::Degraded,
			@@ -498,6 +535,37 @@
498	535		}
499	536		}
500	537
	538	+	/// CA-bundle freshness as a condition.
	539	+	///
	540	+	/// `thin` is the only red: a bundle below the certificate floor cannot validate
	541	+	/// the public web PKI, so outbound TLS on this host is already broken or one
	542	+	/// handshake from it. `stale` (a newer package is available) and `unknown` (the
	543	+	/// package lists are too old for the comparison to mean anything) are drift, and
	544	+	/// drift is yellow. A probe that could not run is yellow for the same reason the
	545	+	/// SyncKit readout is: PoM has stopped being able to answer the question, which
	546	+	/// is worth saying out loud but is not itself an outage.
	547	+	fn ca_bundle_condition(ca: &CaBundleView) -> Condition {
	548	+	let status = match ca.status.as_str() {
	549	+	"ok" => Status::Ok,
	550	+	"thin" => Status::Failed,
	551	+	"stale" \| "unknown" \| "error" => Status::Degraded,
	552	+	_ => Status::Unknown,
	553	+	};
	554	+	let detail = if let Some(error) = &ca.error {
	555	+	format!("probe error: {error}")
	556	+	} else if !ca.issues.is_empty() {
	557	+	ca.issues.join("; ")
	558	+	} else {
	559	+	"trust anchors current".into()
	560	+	};
	561	+	Condition {
	562	+	condition_type: "ca_bundle".into(),
	563	+	status,
	564	+	since: parse_instant(&ca.checked_at),
	565	+	detail: Some(detail),
	566	+	}
	567	+	}
	568	+
501	569		/// The SyncKit field-version readout as a condition. Only two outcomes, and
502	570		/// neither of them depends on which versions came back: `ok` when the reading was
503	571		/// taken (the detail is the distribution), `degraded` when it could not be. No
			@@ -786,6 +854,7 @@
786	854		backups: Vec::new(),
787	855		scan_pipeline: None,
788	856		systemd: None,
	857	+	ca_bundle: None,
789	858		synckit_fleet: None,
790	859		tests: None,
791	860		dns: None,
			@@ -882,6 +951,9 @@
882	951		days_remaining: 9,
883	952		checked_at: checked_at(),
884	953		error: None,
	954	+	webpki_trusted: Some(true),
	955	+	platform_trusted: Some(true),
	956	+	platform_error: None,
885	957		});
886	958		let p = payload(&[t], now());
887	959
			@@ -904,6 +976,9 @@
904	976		days_remaining: -3,
905	977		checked_at: checked_at(),
906	978		error: None,
	979	+	webpki_trusted: Some(true),
	980	+	platform_trusted: Some(true),
	981	+	platform_error: None,
907	982		});
908	983		let p = payload(&[t], now());
909	984
			@@ -930,6 +1005,124 @@
930	1005		days_remaining: 60,
931	1006		checked_at: checked_at(),
932	1007		error: None,
	1008	+	webpki_trusted: Some(true),
	1009	+	platform_trusted: Some(true),
	1010	+	platform_error: None,
	1011	+	});
	1012	+	let p = payload(&[t], now());
	1013	+
	1014	+	let n = node(&p, "target:mnw");
	1015	+	assert_eq!(n.status, Status::Ok);
	1016	+	let tls = n
	1017	+	.conditions
	1018	+	.iter()
	1019	+	.find(\|c\| c.condition_type == "tls")
	1020	+	.unwrap();
	1021	+	assert_eq!(tls.status, Status::Ok);
	1022	+	assert!(tls.detail.as_deref().unwrap().contains("60 days remaining"));
	1023	+	}
	1024	+
	1025	+	#[test]
	1026	+	fn a_host_trust_store_that_rejects_a_publicly_valid_chain_degrades_the_target() {
	1027	+	// multithreaded takes its outbound trust anchors from the host CA
	1028	+	// bundle on every path it has, with no in-binary fallback. A bundle
	1029	+	// that goes stale or thin therefore breaks OAuth, link previews and S3
	1030	+	// at once while the certificates themselves are perfectly good, which
	1031	+	// is why the two stores are reported separately rather than folded.
	1032	+	let mut t = healthy("mnw");
	1033	+	t.tls = Some(TlsView {
	1034	+	valid: true,
	1035	+	days_remaining: 60,
	1036	+	checked_at: checked_at(),
	1037	+	error: None,
	1038	+	webpki_trusted: Some(true),
	1039	+	platform_trusted: Some(false),
	1040	+	platform_error: Some("invalid peer certificate: UnknownIssuer".into()),
	1041	+	});
	1042	+	let p = payload(&[t], now());
	1043	+
	1044	+	let n = node(&p, "target:mnw");
	1045	+	assert_eq!(n.status, Status::Degraded);
	1046	+	let tls = n
	1047	+	.conditions
	1048	+	.iter()
	1049	+	.find(\|c\| c.condition_type == "tls")
	1050	+	.unwrap();
	1051	+	assert_eq!(tls.status, Status::Degraded);
	1052	+	let detail = tls.detail.as_deref().unwrap();
	1053	+	assert!(detail.contains("host trust store"));
	1054	+	assert!(detail.contains("UnknownIssuer"));
	1055	+	}
	1056	+
	1057	+	#[test]
	1058	+	fn a_thin_ca_bundle_fails_the_target_and_a_stale_one_only_degrades_it() {
	1059	+	// The severity split is the point: a bundle below the certificate floor
	1060	+	// means outbound TLS is broken now, while a package one release behind
	1061	+	// is drift. Collapsing them would either page on drift or bury an outage.
	1062	+	let mut thin = healthy("mnw");
	1063	+	thin.ca_bundle = Some(CaBundleView {
	1064	+	status: "thin".into(),
	1065	+	issues: vec!["bundle holds 3 certificates, below the floor of 80".into()],
	1066	+	checked_at: checked_at(),
	1067	+	error: None,
	1068	+	});
	1069	+	assert_eq!(
	1070	+	node(&payload(&[thin], now()), "target:mnw").status,
	1071	+	Status::Failed
	1072	+	);
	1073	+
	1074	+	let mut stale = healthy("mnw");
	1075	+	stale.ca_bundle = Some(CaBundleView {
	1076	+	status: "stale".into(),
	1077	+	issues: vec!["ca-certificates 20260601 installed, 20261101 available".into()],
	1078	+	checked_at: checked_at(),
	1079	+	error: None,
	1080	+	});
	1081	+	let p = payload(&[stale], now());
	1082	+	let n = node(&p, "target:mnw");
	1083	+	assert_eq!(n.status, Status::Degraded);
	1084	+	let ca = n
	1085	+	.conditions
	1086	+	.iter()
	1087	+	.find(\|c\| c.condition_type == "ca_bundle")
	1088	+	.unwrap();
	1089	+	assert!(ca.detail.as_deref().unwrap().contains("20261101 available"));
	1090	+	}
	1091	+
	1092	+	#[test]
	1093	+	fn a_ca_bundle_probe_that_could_not_run_says_so_rather_than_reading_green() {
	1094	+	let mut t = healthy("mnw");
	1095	+	t.ca_bundle = Some(CaBundleView {
	1096	+	status: "error".into(),
	1097	+	issues: Vec::new(),
	1098	+	checked_at: checked_at(),
	1099	+	error: Some("apt-cache policy failed to run: No such file or directory".into()),
	1100	+	});
	1101	+	let p = payload(&[t], now());
	1102	+	let n = node(&p, "target:mnw");
	1103	+	assert_eq!(n.status, Status::Degraded);
	1104	+	let ca = n
	1105	+	.conditions
	1106	+	.iter()
	1107	+	.find(\|c\| c.condition_type == "ca_bundle")
	1108	+	.unwrap();
	1109	+	assert!(ca.detail.as_deref().unwrap().contains("probe error"));
	1110	+	}
	1111	+
	1112	+	#[test]
	1113	+	fn a_pre_migration_tls_row_reports_expiry_and_claims_nothing_about_trust() {
	1114	+	// Rows written before the trust columns existed carry NULL, not false.
	1115	+	// Reading those as "untrusted" would light up every target on the first
	1116	+	// run after an upgrade, which trains the eye to ignore the condition.
	1117	+	let mut t = healthy("mnw");
	1118	+	t.tls = Some(TlsView {
	1119	+	valid: true,
	1120	+	days_remaining: 60,
	1121	+	checked_at: checked_at(),
	1122	+	error: None,
	1123	+	webpki_trusted: None,
	1124	+	platform_trusted: None,
	1125	+	platform_error: None,
933	1126		});
934	1127		let p = payload(&[t], now());
935	1128
			@@ -1038,6 +1231,9 @@
1038	1231		days_remaining: -1,
1039	1232		checked_at: checked_at(),
1040	1233		error: None,
	1234	+	webpki_trusted: Some(true),
	1235	+	platform_trusted: Some(true),
	1236	+	platform_error: None,
1041	1237		});
1042	1238		let p = payload(&[t], now());
1043	1239

M pom/src/types.rs +32

			@@ -33,6 +33,8 @@
33	33		ScanPipelineRecovery,
34	34		SystemdFailure,
35	35		SystemdRecovery,
	36	+	CaBundleStale,
	37	+	CaBundleRecovery,
36	38		MonitoringOffline,
37	39		MonitoringRecovery,
38	40		}
			@@ -64,6 +66,8 @@
64	66		Self::ScanPipelineRecovery => write!(f, "scan_pipeline_recovery"),
65	67		Self::SystemdFailure => write!(f, "systemd_failure"),
66	68		Self::SystemdRecovery => write!(f, "systemd_recovery"),
	69	+	Self::CaBundleStale => write!(f, "ca_bundle_stale"),
	70	+	Self::CaBundleRecovery => write!(f, "ca_bundle_recovery"),
67	71		Self::MonitoringOffline => write!(f, "monitoring_offline"),
68	72		Self::MonitoringRecovery => write!(f, "monitoring_recovery"),
69	73		}
			@@ -98,6 +102,8 @@
98	102		"scan_pipeline_recovery" => Ok(Self::ScanPipelineRecovery),
99	103		"systemd_failure" => Ok(Self::SystemdFailure),
100	104		"systemd_recovery" => Ok(Self::SystemdRecovery),
	105	+	"ca_bundle_stale" => Ok(Self::CaBundleStale),
	106	+	"ca_bundle_recovery" => Ok(Self::CaBundleRecovery),
101	107		"monitoring_offline" => Ok(Self::MonitoringOffline),
102	108		"monitoring_recovery" => Ok(Self::MonitoringRecovery),
103	109		other => Err(format!("unknown alert category: {other}")),
			@@ -579,6 +585,32 @@
579	585		pub error: Option<String>,
580	586		}
581	587
	588	+	#[derive(Debug, Clone, Serialize, Deserialize)]
	589	+	pub struct CaBundleCheckResult {
	590	+	/// Config key identifying the monitored host target.
	591	+	pub target: String,
	592	+	/// Overall status: "ok", "stale", "thin", "unknown", or "error".
	593	+	pub status: String,
	594	+	/// Package providing the trust anchors (e.g. "ca-certificates").
	595	+	pub package: String,
	596	+	/// Version currently installed, as dpkg reports it.
	597	+	pub installed: Option<String>,
	598	+	/// Version apt would install now. Equal to `installed` on a current host.
	599	+	pub candidate: Option<String>,
	600	+	/// Certificates counted in the concatenated PEM bundle.
	601	+	pub cert_count: Option<i64>,
	602	+	/// Age in whole hours of the last successful package-list update. `None`
	603	+	/// when the stamp file is absent, which is itself reported.
	604	+	pub lists_age_hours: Option<i64>,
	605	+	/// One short line per fired condition, in the order they were evaluated.
	606	+	pub issues: Vec<String>,
	607	+	/// When this check was performed, in RFC 3339 format (UTC).
	608	+	pub checked_at: String,
	609	+	/// Error message if the probe itself could not run. `None` when the probe
	610	+	/// worked and merely found something wrong (that goes in `issues`).
	611	+	pub error: Option<String>,
	612	+	}
	613	+
582	614		#[derive(Debug, Clone, Serialize, Deserialize)]
583	615		pub struct BackupCheckResult {
584	616		/// Config key identifying the monitored target.

M pom/tests/integration.rs +17 -13

			@@ -413,7 +413,7 @@
413	413		// A fresh in-memory DB should run all migrations and reach the latest version.
414	414		let pool = db::connect_in_memory().await.unwrap();
415	415		let version = db::get_schema_version(&pool).await.unwrap();
416		-	assert_eq!(version, 13);
	416	+	assert_eq!(version, 15);
417	417
418	418		// Verify the schema_version table has entries for each migration
419	419		let rows = sqlx::query_as::<_, (i64, String)>(
			@@ -422,7 +422,7 @@
422	422		.fetch_all(&pool)
423	423		.await
424	424		.unwrap();
425		-	assert_eq!(rows.len(), 13);
	425	+	assert_eq!(rows.len(), 15);
426	426		assert_eq!(rows[0].0, 1);
427	427		assert_eq!(rows[0].1, "initial schema");
428	428		assert_eq!(rows[1].0, 2);
			@@ -449,6 +449,10 @@
449	449		assert_eq!(rows[11].1, "add systemd_checks table");
450	450		assert_eq!(rows[12].0, 13);
451	451		assert_eq!(rows[12].1, "add synckit_fleet_checks table");
	452	+	assert_eq!(rows[13].0, 14);
	453	+	assert_eq!(rows[13].1, "record per-trust-store results on tls_checks");
	454	+	assert_eq!(rows[14].0, 15);
	455	+	assert_eq!(rows[14].1, "add ca_bundle_checks table");
452	456
453	457		// Verify actual tables were created by inserting data
454	458		let snapshot = HealthSnapshot {
			@@ -468,18 +472,18 @@
468	472		async fn migration_already_current_is_idempotent() {
469	473		// Running migrations on an already-migrated DB should be a no-op.
470	474		let pool = db::connect_in_memory().await.unwrap();
471		-	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 13);
	475	+	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 15);
472	476
473	477		// Run migrations again
474	478		db::run_migrations(&pool).await.unwrap();
475		-	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 13);
	479	+	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 15);
476	480
477	481		// schema_version should still have exactly thirteen entries (not duplicated)
478	482		let count = sqlx::query_as::<_, (i64,)>("SELECT COUNT(*) FROM schema_version")
479	483		.fetch_one(&pool)
480	484		.await
481	485		.unwrap();
482		-	assert_eq!(count.0, 13);
	486	+	assert_eq!(count.0, 15);
483	487		}
484	488
485	489		#[tokio::test]
			@@ -535,11 +539,11 @@
535	539		.await
536	540		.unwrap();
537	541
538		-	// Now run migrations, should detect existing tables, stamp as v1, then run v2+v3+v4+v5+v6
	542	+	// Now run migrations, should detect existing tables, stamp as v1, then run v2..v15
539	543		db::run_migrations(&pool).await.unwrap();
540	544
541		-	// Version should be 13 (stamped v1 + ran v2..v13)
542		-	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 13);
	545	+	// Version should be 15 (stamped v1 + ran v2..v15)
	546	+	assert_eq!(db::get_schema_version(&pool).await.unwrap(), 15);
543	547
544	548		// Description should indicate pre-existing
545	549		let row =
			@@ -860,7 +864,7 @@
860	864		async fn migration_v2_creates_alerts_table() {
861	865		let pool = db::connect_in_memory().await.unwrap();
862	866		let version = db::get_schema_version(&pool).await.unwrap();
863		-	assert_eq!(version, 13);
	867	+	assert_eq!(version, 15);
864	868
865	869		// Verify alerts table exists by inserting
866	870		let id = db::insert_alert(
			@@ -946,7 +950,7 @@
946	950		async fn migration_v3_creates_tls_checks_table() {
947	951		let pool = db::connect_in_memory().await.unwrap();
948	952		let version = db::get_schema_version(&pool).await.unwrap();
949		-	assert_eq!(version, 13);
	953	+	assert_eq!(version, 15);
950	954
951	955		// Verify tls_checks table exists by inserting
952	956		let status = pom::types::TlsStatus {
			@@ -1184,7 +1188,7 @@
1184	1188		async fn migration_v4_creates_incidents_table() {
1185	1189		let pool = db::connect_in_memory().await.unwrap();
1186	1190		let version = db::get_schema_version(&pool).await.unwrap();
1187		-	assert_eq!(version, 13);
	1191	+	assert_eq!(version, 15);
1188	1192
1189	1193		// Verify incidents table exists by inserting
1190	1194		let id = db::insert_incident(&pool, "mnw", "operational", "degraded")
			@@ -1324,7 +1328,7 @@
1324	1328		async fn migration_v5_creates_route_checks_table() {
1325	1329		let pool = db::connect_in_memory().await.unwrap();
1326	1330		let version = db::get_schema_version(&pool).await.unwrap();
1327		-	assert_eq!(version, 13);
	1331	+	assert_eq!(version, 15);
1328	1332
1329	1333		// Verify route_checks table exists by inserting
1330	1334		let result = pom::checks::routes::RouteCheckResult {
			@@ -2473,7 +2477,7 @@
2473	2477		async fn migration_v6_creates_dns_and_whois_tables() {
2474	2478		let pool = db::connect_in_memory().await.unwrap();
2475	2479		let version = db::get_schema_version(&pool).await.unwrap();
2476		-	assert_eq!(version, 13);
	2480	+	assert_eq!(version, 15);
2477	2481
2478	2482		// Verify dns_checks table exists
2479	2483		let dns_result = DnsCheckResult {

M pom/src/alerts/mod.rs +10 -6

			@@ -11,6 +11,7 @@
11	11		use crate::types::AlertCategory;
12	12
13	13		mod backup;
	14	+	mod ca_bundle;
14	15		mod cors;
15	16		mod dns;
16	17		mod health;
			@@ -82,15 +83,18 @@
82	83		/// rejects the push with 422).
83	84		fn mnw_kind(category: AlertCategory) -> &'static str {
84	85		use AlertCategory::{
85		-	BackupRecovery, BackupStale, CorsFailure, CorsRecovery, DnsMismatch, DnsRecovery, Health,
86		-	LatencyDrift, LatencyRecovery, MonitoringOffline, MonitoringRecovery, PeerMissing,
87		-	PeerRecovery, Recovery, RouteFailure, RouteRecovery, ScanPipelineDegraded,
88		-	ScanPipelineRecovery, SystemdFailure, SystemdRecovery, TestDurationDrift, TlsError,
89		-	TlsExpiry, TlsRecovery, WhoisError, WhoisExpiry,
	86	+	BackupRecovery, BackupStale, CaBundleRecovery, CaBundleStale, CorsFailure, CorsRecovery,
	87	+	DnsMismatch, DnsRecovery, Health, LatencyDrift, LatencyRecovery, MonitoringOffline,
	88	+	MonitoringRecovery, PeerMissing, PeerRecovery, Recovery, RouteFailure, RouteRecovery,
	89	+	ScanPipelineDegraded, ScanPipelineRecovery, SystemdFailure, SystemdRecovery,
	90	+	TestDurationDrift, TlsError, TlsExpiry, TlsRecovery, WhoisError, WhoisExpiry,
90	91		};
91	92		match category {
92	93		Health \| Recovery => "health",
93		-	TlsExpiry \| TlsError \| TlsRecovery => "tls",
	94	+	// The host CA bundle folds onto the "tls" domain: it is a trust-anchor
	95	+	// problem, and every symptom it produces is a failed handshake. Adding a
	96	+	// domain would need a matching MNW `AlertKind` or the ingest 422s.
	97	+	TlsExpiry \| TlsError \| TlsRecovery \| CaBundleStale \| CaBundleRecovery => "tls",
94	98		DnsMismatch \| DnsRecovery => "dns",
95	99		WhoisExpiry \| WhoisError => "whois",
96	100		LatencyDrift \| LatencyRecovery \| TestDurationDrift => "latency",

M pom/src/checks/mod.rs +3 -2

			@@ -1,8 +1,9 @@
1	1		//! Check implementations, one module per probe kind: health, TLS, DNS, WHOIS,
2		-	//! routes, CORS, backups, SSH, port scans, local systemd units, the SyncKit
3		-	//! field-version readout, plus latency-drift analysis.
	2	+	//! routes, CORS, backups, SSH, port scans, local systemd units, the local CA
	3	+	//! bundle, the SyncKit field-version readout, plus latency-drift analysis.
4	4
5	5		pub mod backup;
	6	+	pub mod ca_bundle;
6	7		pub mod cors;
7	8		pub mod dns;
8	9		pub mod drift;

M pom/src/cli/serve.rs +6

			@@ -115,6 +115,12 @@
115	115		&token,
116	116		alerter.as_ref(),
117	117		));
	118	+	handles.extend(tasks::spawn_ca_bundle_tasks(
	119	+	config,
	120	+	pool,
	121	+	&token,
	122	+	alerter.as_ref(),
	123	+	));
118	124		// No alerter: the fleet readout has nothing to alert on. See the task module.
119	125		handles.extend(tasks::spawn_synckit_fleet_tasks(config, pool, &token));
120	126		handles.push(tasks::spawn_prune_task(pool, prune_days, &token));

M pom/src/db/maintenance.rs +8

M pom/src/db/migrations.rs +39

M pom/src/db/mod.rs +6 -3

M pom/src/db/tls.rs +16 -3

M pom/src/cli/tasks/mod.rs +2

A pom/src/alerts/ca_bundle.rs +89

A pom/src/checks/ca_bundle.rs +394

A pom/src/cli/tasks/ca_bundle.rs +92

A pom/src/db/ca_bundle.rs +71