| 1 |
# PoM Architecture |
| 2 |
|
| 3 |
## System Overview |
| 4 |
|
| 5 |
PoM runs in three modes, selected by how it is invoked: |
| 6 |
|
| 7 |
1. **CLI mode** (`pom health`, `pom test`, `pom status`, etc.): runs a single command and exits. Useful for ad-hoc checks and cron jobs. |
| 8 |
2. **Serve mode** (`pom serve`): long-running daemon that spawns per-target health check loops, TLS check loops, peer heartbeat tasks, a daily prune task, and an HTTP API server. This is the production deployment mode. |
| 9 |
3. **MCP server mode** (bare `pom` with no subcommand): launches an MCP server over stdio for Claude integration. Exposes health checks, test execution, history queries, and mesh status as MCP tools, plus a read-only set (`status_table`, `target_status`, `versions`, `incidents`, `trends`) for orienting on what is live. |
| 10 |
|
| 11 |
The read-only tools take an optional `instance`. Omitted, they read this machine's database directly and need no running daemon. Named, they resolve a configured peer and read that instance's HTTP API, which is the only way to see checks that are local to that host: systemd units and backup freshness on the production box are not observable from anywhere else. Both paths produce the same types, so the formatting is written once. |
| 12 |
|
| 13 |
All three modes load the same TOML config and connect to the same SQLite database. |
| 14 |
|
| 15 |
## Module Map |
| 16 |
|
| 17 |
|
| 18 |
|
| 19 |
| `main` | `src/main.rs` | Entry point: parses CLI args, dispatches to CLI handler or MCP server | |
| 20 |
| `cli` | `src/cli/` | CLI command handlers. `mod.rs` dispatch, plus `serve.rs`, `status.rs`, `incident.rs`, `transition.rs`, and `tasks/` (one spawner per background loop: health, tls, dns, whois, routes, cors, backup, systemd, ca_bundle, scan_pipeline, synckit_fleet, meta_alert, prune) | |
| 21 |
| `config` | `src/config.rs` | TOML config loading, types for targets/peers/alerts/serve settings | |
| 22 |
| `types` | `src/types.rs` | Shared domain types: HealthSnapshot, TestRun, TlsStatus, LatencyStats, TestStaleness | |
| 23 |
| `db` | `src/db/` | SQLite persistence. `migrations.rs` holds the numbered schema migrations and pool construction; the rest is one query module per subject: health, test_runs, alerts, incidents, peers, tls, dns, whois, routes, cors, backup, systemd, ca_bundle, scan_pipeline, synckit_fleet, maintenance | |
| 24 |
| `api` | `src/api.rs` | Axum HTTP API: status, trends, peer info, mesh view, bearer token auth middleware | |
| 25 |
| `alerts` | `src/alerts/` | Alerter struct: sends emails via Postmark on status transitions, with cooldown tracking. One module per alert kind: health, tls, latency, offline, peer, dns, whois, route, cors, backup, systemd, ca_bundle, scan, test_duration | |
| 26 |
| `peer` | `src/peer.rs` | Peer mesh: identity management, heartbeat loops, grace period state machine, mesh state | |
| 27 |
| `status` | `src/status.rs` | PoM's projection onto the shared operator status payload | |
| 28 |
| `dashboard` | `src/dashboard.rs` | Optional HTML dashboard served at `GET /` | |
| 29 |
| `tls` | `src/tls.rs` | Shared TLS configuration for pom's outbound HTTP, used by every probe | |
| 30 |
| `display` | `src/display.rs` | Pure formatting functions for CLI output (no I/O) | |
| 31 |
| `versions` | `src/versions.rs` | The `pom versions` roll-up: live version/sha per target, plus commits behind the local checkout | |
| 32 |
| `error` | `src/error.rs` | Typed error enum (PomError) wrapping IO, DB, HTTP, JSON, config errors | |
| 33 |
| `checks::http` | `src/checks/http.rs` | HTTP health probe: issues the request, applies configured JSON expectations, classifies the response into a HealthStatus | |
| 34 |
| `checks::drift` | `src/checks/drift.rs` | Trend and staleness analysis over stored check history: latency drift, test duration, test staleness | |
| 35 |
| `checks::tls` | `src/checks/tls.rs` | TLS certificate prober: connect, inspect the leaf cert, track expiry | |
| 36 |
| `checks::ssh` | `src/checks/ssh.rs` | Remote test runner: validates the filter, runs the target's command over SSH, parses output into a TestRun | |
| 37 |
| `checks::ssh_banner` | `src/checks/ssh_banner.rs` | TCP connect and verify the SSH protocol banner | |
| 38 |
| `checks::parse` | `src/checks/parse.rs` | CI output parser: extracts PASS/FAIL steps and cargo test counts | |
| 39 |
| `checks::dns` | `src/checks/dns.rs` | DNS record verification: resolves hostnames, compares against expected values | |
| 40 |
| `checks::whois` | `src/checks/whois.rs` | WHOIS domain expiry checking over raw TCP | |
| 41 |
| `checks::routes` | `src/checks/routes.rs` | Route accessibility: verifies expected pages are reachable | |
| 42 |
| `checks::cors` | `src/checks/cors.rs` | CORS preflight: sends OPTIONS and checks Access-Control headers | |
| 43 |
| `checks::backup` | `src/checks/backup.rs` | Local filesystem backup verification: scans for PostgreSQL backup files, checks recency | |
| 44 |
| `checks::systemd` | `src/checks/systemd.rs` | Local systemd unit health: liveness, crash-loops, optionally any failed unit on the host | |
| 45 |
| `checks::ca_bundle` | `src/checks/ca_bundle.rs` | Local CA-bundle freshness: installed vs candidate package version, age of the last package-list update, certificate count against a floor | |
| 46 |
| `checks::scan_pipeline` | `src/checks/scan_pipeline.rs` | Polls a makenotwork instance's upload scan health and applies the audit thresholds | |
| 47 |
| `checks::synckit_fleet` | `src/checks/synckit_fleet.rs` | Polls a makenotwork instance for which SyncKit SDK versions are syncing | |
| 48 |
| `tools` | `src/tools/mod.rs` | MCP server definition (PomServer), tool registration via rmcp | |
| 49 |
| `tools::health` | `src/tools/health.rs` | MCP tool implementations for health checks, history, targets, mesh status | |
| 50 |
| `tools::tests` | `src/tools/tests.rs` | MCP tool implementations for test execution, history, raw output | |
| 51 |
| `tools::orient` | `src/tools/orient.rs` | Read-only MCP tools (status_table, target_status, versions, incidents, trends), local or against a peer | |
| 52 |
|
| 53 |
## Data Flow |
| 54 |
|
| 55 |
``` |
| 56 |
pom.toml (config) |
| 57 |
| |
| 58 |
v |
| 59 |
Config::load() --> targets, peers, alerts, serve settings |
| 60 |
| |
| 61 |
v |
| 62 |
db::connect() --> SQLite pool (WAL mode, versioned migrations) |
| 63 |
| |
| 64 |
+---> [CLI mode] single command --> check/query --> display --> exit |
| 65 |
| |
| 66 |
+---> [Serve mode] |
| 67 |
| | |
| 68 |
| +--> per-target health check loop (configurable interval) |
| 69 |
| | check_health() --> insert_health_check() |
| 70 |
| | compare with previous --> alert on transition |
| 71 |
| | detect latency drift --> alert if sustained |
| 72 |
| | open/close incidents on status changes |
| 73 |
| | |
| 74 |
| +--> per-target TLS check loop (hourly default) |
| 75 |
| | check_tls() --> insert_tls_check() |
| 76 |
| | alert on expiry warning or error |
| 77 |
| | |
| 78 |
| +--> per-peer heartbeat loop (60s default) |
| 79 |
| | GET /api/peer/info --> update mesh state |
| 80 |
| | GET /api/peer/status --> cache for mesh view |
| 81 |
| | grace period state machine on failure |
| 82 |
| | |
| 83 |
| +--> daily prune task (configurable retention) |
| 84 |
| | |
| 85 |
| +--> HTTP API server (Axum, configurable bind address) |
| 86 |
| |
| 87 |
+---> [MCP mode] stdio transport --> tool calls --> same check/query logic |
| 88 |
``` |
| 89 |
|
| 90 |
## Peer Mesh Design |
| 91 |
|
| 92 |
Each PoM instance has a persistent UUID (stored at `~/.local/share/pom/instance_id`). Peers are configured by name with an address, a `on_missing` policy, and an optional grace count. |
| 93 |
|
| 94 |
### Heartbeat State Machine |
| 95 |
|
| 96 |
``` |
| 97 |
Unknown --> (success) --> Online |
| 98 |
Unknown --> (failure) --> GracePeriod --> (failures >= grace_count) --> Missing |
| 99 |
Online --> (failure) --> GracePeriod --> (failures >= grace_count) --> Missing |
| 100 |
Missing --> (success) --> Online (triggers recovery alert) |
| 101 |
``` |
| 102 |
|
| 103 |
Each heartbeat cycle: |
| 104 |
1. GET `/api/peer/info`: verifies identity, measures latency |
| 105 |
2. On first contact, store the peer's UUID in `peer_identities` table |
| 106 |
3. On subsequent contacts, reject UUID mismatches (prevents impersonation) |
| 107 |
4. GET `/api/peer/status`: caches the peer's full status for mesh aggregation |
| 108 |
5. Record heartbeat result in `peer_heartbeats` table |
| 109 |
|
| 110 |
### On Missing Policy |
| 111 |
|
| 112 |
- `alert`: send email alert when peer transitions to Missing, send recovery when it returns |
| 113 |
- `log`: log the event, no email |
| 114 |
- `ignore`: suppress entirely |
| 115 |
|
| 116 |
## Database Schema |
| 117 |
|
| 118 |
SQLite with WAL journal mode. Schema is managed through numbered migrations (currently v1-v13). |
| 119 |
|
| 120 |
### Where the database lives |
| 121 |
|
| 122 |
`storage.db_path` in the config file, when set, and it must be absolute. Unset, the path is read out of the XDG data directory, which is where the daemon and the CLI on the same host once disagreed: `pom.service` sets `XDG_DATA_HOME=/var/lib` and an interactive login does not, so `pom serve` wrote `/var/lib/pom/pom.db` while `pom test` run by hand wrote `/var/lib/pom/.local/share/pom/pom.db`. The suites ran, passed, and printed green into a database nothing served. Any instance whose database is not in the invoking user's own `~/.local/share` should set the path; running unconfigured warns. |
| 123 |
|
| 124 |
Opening a database that does not exist is an error naming the path. `--init` creates it, and is what a first install runs. Auto-creating on open is what turned one wrong path into a silent second store rather than a complaint. |
| 125 |
|
| 126 |
The instance ID sits beside the database, in whichever directory that resolves to. |
| 127 |
|
| 128 |
### Tables |
| 129 |
|
| 130 |
|
| 131 |
|
| 132 |
| `schema_version` | Migration tracking | version, description, applied_at | |
| 133 |
| `health_checks` | HTTP health check results | target, status, checked_at, response_time_ms, details_json, error | |
| 134 |
| `test_runs` | SSH test execution results | target, started_at, duration_secs, exit_code, passed, summary_json, raw_output | |
| 135 |
| `peer_identities` | First-seen peer UUIDs | peer_name (PK), instance_id, first_seen | |
| 136 |
| `peer_heartbeats` | Heartbeat history | peer_name, status, latency_ms, checked_at | |
| 137 |
| `alerts` | Alert history + cooldown tracking | target, alert_type, from_status, to_status, sent_at | |
| 138 |
| `tls_checks` | TLS certificate probe results | target, host, valid, days_remaining, not_before, not_after, subject, issuer | |
| 139 |
| `incidents` | Health incidents (open/closed) | target, started_at, ended_at, duration_secs, from_status, to_status | |
| 140 |
| `route_checks` | Per-path HTTP route probe results | target, path, status_code, ok, response_time_ms, checked_at, error | |
| 141 |
| `dns_checks` | DNS record expectation checks | target, name, record_type, expected, actual, matches, checked_at, error | |
| 142 |
| `whois_checks` | Domain registration and expiry probes | target, domain, registrar, expiry_date, days_remaining, nameservers, checked_at, error | |
| 143 |
| `test_details` | Per-test rows for a run in `test_runs` | run_id, test_name, passed, duration_ms | |
| 144 |
| `cors_checks` | CORS preflight probe results | target, url, origin, method, passes, checked_at, error | |
| 145 |
| `backup_checks` | Backup freshness and size probes | target, database_name, status, last_backup_at, size_bytes, age_hours, checked_at, error | |
| 146 |
| `pending_alerts` | Alert outbox with retry state | alert_key, category, channel, subject, body, priority, source, attempts, next_retry_at | |
| 147 |
| `scan_pipeline_checks` | Upload scan queue depth and stuck-job counts | target, status, issues, queue_pending, queue_running, queue_stuck, held_total, checked_at | |
| 148 |
| `systemd_checks` | systemd unit state on a host | target, status, units, failed_units, issues, checked_at, error | |
| 149 |
| `ca_bundle_checks` | Host trust-anchor freshness | target, status, package, installed, candidate, cert_count, lists_age_hours, issues, checked_at, error | |
| 150 |
| `synckit_fleet_checks` | SyncKit device and version spread over a window | target, window_days, devices, versions, checked_at, error | |
| 151 |
|
| 152 |
Pre-migration databases are detected by the presence of the `health_checks` table and stamped as v1 without re-running the initial migration. |
| 153 |
|
| 154 |
## Alert Pipeline |
| 155 |
|
| 156 |
``` |
| 157 |
Health status change detected (previous != current) |
| 158 |
| |
| 159 |
+--> operational -> non-operational: send_health_alert(), open incident |
| 160 |
+--> non-operational -> operational: send_health_recovery(), close incidents |
| 161 |
+--> non-operational -> different non-operational: close old incident, open new, alert |
| 162 |
| |
| 163 |
TLS check detects issue |
| 164 |
| |
| 165 |
+--> was OK, now invalid/error: send_tls_error_alert() |
| 166 |
+--> was OK, now within warn_days: send_tls_expiry_alert() |
| 167 |
+--> was bad, now OK: send_tls_recovery() |
| 168 |
| |
| 169 |
Latency drift detected (all recent checks exceed baseline * threshold) |
| 170 |
| |
| 171 |
+--> entered drift: send_latency_drift_alert() |
| 172 |
+--> exited drift: send_latency_recovery() |
| 173 |
| |
| 174 |
Peer transitions to Missing |
| 175 |
| |
| 176 |
+--> send_peer_missing() (if on_missing = alert) |
| 177 |
+--> peer recovers: send_peer_recovery() |
| 178 |
``` |
| 179 |
|
| 180 |
All alerts except recoveries are subject to a per-target cooldown (default 300s). Recoveries always send immediately. Without a Postmark token, alerts are logged to stdout (dev mode). |
| 181 |
|
| 182 |
## API Endpoints |
| 183 |
|
| 184 |
All endpoints require `Authorization: Bearer <token>` when `api_token` is configured (in config or via `POM_API_TOKEN` env var). Without a token configured, all requests pass through. |
| 185 |
|
| 186 |
|
| 187 |
|
| 188 |
| `/api/status` | GET | JSON summary of all targets (latest health, uptime, latency, TLS, staleness, incidents) | |
| 189 |
| `/api/status/{target}` | GET | Same as above for a single target | |
| 190 |
| `/api/trends/{target}` | GET | Latency trend data with configurable window and bucket size (`?hours=24&bucket_minutes=60`) | |
| 191 |
| `/api/versions` | GET | What each target is running: version, git sha, when that version was first seen, when it was last checked, commits behind this host's checkout | |
| 192 |
| `/api/peer/info` | GET | This instance's identity (id, name, version, targets, started_at) | |
| 193 |
| `/api/peer/status` | GET | This instance's full view: identity + target statuses + peer summaries | |
| 194 |
| `/api/mesh` | GET | Aggregated mesh view: self + each peer's cached status | |
| 195 |
|
| 196 |
## Check Types |
| 197 |
|
| 198 |
### HTTP Health Check |
| 199 |
|
| 200 |
Sends GET to the target's health URL. Classifies the response: |
| 201 |
- JSON with `"status": "operational"` --> Operational |
| 202 |
- JSON with `"status": "degraded"` --> Degraded |
| 203 |
- Non-JSON 2xx --> Degraded |
| 204 |
- Non-2xx or unknown status --> Error |
| 205 |
- Connection failure --> Unreachable |
| 206 |
|
| 207 |
Extracts version, git_sha, uptime, checks, and monitoring from the JSON response body. A target that reports `git_sha` gets a commits-behind figure in `pom versions`; one that does not still reports its version. Supports expectation validation: expected status code, required body substrings, and JSON field value assertions (with dot-path traversal for nested fields). |
| 208 |
|
| 209 |
### TLS Certificate Check |
| 210 |
|
| 211 |
Connects to host:port, completes a TLS handshake using the system trust store (webpki-roots), extracts the leaf certificate, and parses it with x509-parser. Records validity, days remaining, not_before/not_after, subject, and issuer. Alerts when days_remaining falls below the configured `warn_days` threshold (default 14). |
| 212 |
|
| 213 |
### SSH Test Runner |
| 214 |
|
| 215 |
Executes a configured command on a remote host via `ssh -o BatchMode=yes`. The command string comes from config (today, `cargo test --workspace` run against a staging checkout). Supports an optional filter argument (validated to `[a-zA-Z0-9_:-]` to prevent injection). Output is parsed for PASS/FAIL step lines and `test result:` cargo test summary lines. |
| 216 |
|
| 217 |
## Key Design Decisions |
| 218 |
|
| 219 |
**SQLite over PostgreSQL.** PoM is a single-binary tool that runs on each monitoring host. SQLite keeps it self-contained with zero external dependencies. WAL mode provides concurrent reads during serve mode. Data volume is modest (a few checks per minute, pruned after 30 days). |
| 220 |
|
| 221 |
**Peer mesh over centralized monitoring.** Two independent instances cross-check each other. If the Hetzner instance goes down, Astra detects it (and vice versa). No single point of failure for the monitoring layer itself. |
| 222 |
|
| 223 |
**Bearer token auth.** Simple, stateless, sufficient for machine-to-machine API access between peers. Configured per-peer and per-instance. No user management needed. |
| 224 |
|
| 225 |
**Versioned migrations.** The migration system detects pre-migration databases and stamps them without re-running. Each migration is a numbered SQL block. This avoids external migration tools while keeping schema evolution safe. |
| 226 |
|
| 227 |
**Separate check intervals.** Health checks can have per-target interval overrides. TLS checks run on a longer interval (hourly default) since certificate state changes slowly. Peer heartbeats run on a short interval (60s default) for timely failure detection. |
| 228 |
|
| 229 |
**Cooldown on alerts.** Prevents alert storms during flapping. Recovery alerts bypass cooldown so operators always know when a service comes back. |
| 230 |
|