| 1 |
# PoM Architecture |
| 2 |
|
| 3 |
## System Overview |
| 4 |
|
| 5 |
PoM runs in three modes, selected by how it is invoked: |
| 6 |
|
| 7 |
1. **CLI mode** (`pom health`, `pom test`, `pom status`, etc.): runs a single command and exits. Useful for ad-hoc checks and cron jobs. |
| 8 |
2. **Serve mode** (`pom serve`): long-running daemon that spawns per-target health check loops, TLS check loops, peer heartbeat tasks, a daily prune task, and an HTTP API server. This is the production deployment mode. |
| 9 |
3. **MCP server mode** (bare `pom` with no subcommand): launches an MCP server over stdio for Claude integration. Exposes health checks, test execution, history queries, and mesh status as MCP tools, plus a read-only set (`status_table`, `target_status`, `versions`, `incidents`, `trends`) for orienting on what is live. |
| 10 |
|
| 11 |
The read-only tools take an optional `instance`. Omitted, they read this machine's database directly and need no running daemon. Named, they resolve a configured peer and read that instance's HTTP API, which is the only way to see checks that are local to that host: systemd units and backup freshness on the production box are not observable from anywhere else. Both paths produce the same types, so the formatting is written once. |
| 12 |
|
| 13 |
All three modes load the same TOML config and connect to the same SQLite database. |
| 14 |
|
| 15 |
## Module Map |
| 16 |
|
| 17 |
|
| 18 |
|
| 19 |
| `main` | `src/main.rs` | Entry point: parses CLI args, dispatches to CLI handler or MCP server | |
| 20 |
| `cli` | `src/cli/` | CLI command handlers. `mod.rs` dispatch, plus `serve.rs`, `status.rs`, `incident.rs`, `transition.rs`, and `tasks/` (one spawner per background loop: health, tls, dns, whois, routes, cors, backup, systemd, scan_pipeline, synckit_fleet, meta_alert, prune) | |
| 21 |
| `config` | `src/config.rs` | TOML config loading, types for targets/peers/alerts/serve settings | |
| 22 |
| `types` | `src/types.rs` | Shared domain types: HealthSnapshot, TestRun, TlsStatus, LatencyStats, TestStaleness | |
| 23 |
| `db` | `src/db/` | SQLite persistence. `migrations.rs` holds the numbered schema migrations and pool construction; the rest is one query module per subject: health, test_runs, alerts, incidents, peers, tls, dns, whois, routes, cors, backup, systemd, scan_pipeline, synckit_fleet, maintenance | |
| 24 |
| `api` | `src/api.rs` | Axum HTTP API: status, trends, peer info, mesh view, bearer token auth middleware | |
| 25 |
| `alerts` | `src/alerts/` | Alerter struct: sends emails via Postmark on status transitions, with cooldown tracking. One module per alert kind: health, tls, latency, offline, peer, dns, whois, route, cors, backup, systemd, scan, test_duration | |
| 26 |
| `peer` | `src/peer.rs` | Peer mesh: identity management, heartbeat loops, grace period state machine, mesh state | |
| 27 |
| `status` | `src/status.rs` | PoM's projection onto the shared operator status payload | |
| 28 |
| `dashboard` | `src/dashboard.rs` | Optional HTML dashboard served at `GET /` | |
| 29 |
| `tls` | `src/tls.rs` | Shared TLS configuration for pom's outbound HTTP, used by every probe | |
| 30 |
| `display` | `src/display.rs` | Pure formatting functions for CLI output (no I/O) | |
| 31 |
| `versions` | `src/versions.rs` | The `pom versions` roll-up: live version/sha per target, plus commits behind the local checkout | |
| 32 |
| `error` | `src/error.rs` | Typed error enum (PomError) wrapping IO, DB, HTTP, JSON, config errors | |
| 33 |
| `checks::http` | `src/checks/http.rs` | HTTP health probe: issues the request, applies configured JSON expectations, classifies the response into a HealthStatus | |
| 34 |
| `checks::drift` | `src/checks/drift.rs` | Trend and staleness analysis over stored check history: latency drift, test duration, test staleness | |
| 35 |
| `checks::tls` | `src/checks/tls.rs` | TLS certificate prober: connect, inspect the leaf cert, track expiry | |
| 36 |
| `checks::ssh` | `src/checks/ssh.rs` | Remote test runner: validates the filter, runs the target's command over SSH, parses output into a TestRun | |
| 37 |
| `checks::ssh_banner` | `src/checks/ssh_banner.rs` | TCP connect and verify the SSH protocol banner | |
| 38 |
| `checks::parse` | `src/checks/parse.rs` | CI output parser: extracts PASS/FAIL steps and cargo test counts | |
| 39 |
| `checks::dns` | `src/checks/dns.rs` | DNS record verification: resolves hostnames, compares against expected values | |
| 40 |
| `checks::whois` | `src/checks/whois.rs` | WHOIS domain expiry checking over raw TCP | |
| 41 |
| `checks::routes` | `src/checks/routes.rs` | Route accessibility: verifies expected pages are reachable | |
| 42 |
| `checks::cors` | `src/checks/cors.rs` | CORS preflight: sends OPTIONS and checks Access-Control headers | |
| 43 |
| `checks::backup` | `src/checks/backup.rs` | Local filesystem backup verification: scans for PostgreSQL backup files, checks recency | |
| 44 |
| `checks::systemd` | `src/checks/systemd.rs` | Local systemd unit health: liveness, crash-loops, optionally any failed unit on the host | |
| 45 |
| `checks::scan_pipeline` | `src/checks/scan_pipeline.rs` | Polls a makenotwork instance's upload scan health and applies the audit thresholds | |
| 46 |
| `checks::synckit_fleet` | `src/checks/synckit_fleet.rs` | Polls a makenotwork instance for which SyncKit SDK versions are syncing | |
| 47 |
| `tools` | `src/tools/mod.rs` | MCP server definition (PomServer), tool registration via rmcp | |
| 48 |
| `tools::health` | `src/tools/health.rs` | MCP tool implementations for health checks, history, targets, mesh status | |
| 49 |
| `tools::tests` | `src/tools/tests.rs` | MCP tool implementations for test execution, history, raw output | |
| 50 |
| `tools::orient` | `src/tools/orient.rs` | Read-only MCP tools (status_table, target_status, versions, incidents, trends), local or against a peer | |
| 51 |
|
| 52 |
## Data Flow |
| 53 |
|
| 54 |
``` |
| 55 |
pom.toml (config) |
| 56 |
| |
| 57 |
v |
| 58 |
Config::load() --> targets, peers, alerts, serve settings |
| 59 |
| |
| 60 |
v |
| 61 |
db::connect() --> SQLite pool (WAL mode, versioned migrations) |
| 62 |
| |
| 63 |
+---> [CLI mode] single command --> check/query --> display --> exit |
| 64 |
| |
| 65 |
+---> [Serve mode] |
| 66 |
| | |
| 67 |
| +--> per-target health check loop (configurable interval) |
| 68 |
| | check_health() --> insert_health_check() |
| 69 |
| | compare with previous --> alert on transition |
| 70 |
| | detect latency drift --> alert if sustained |
| 71 |
| | open/close incidents on status changes |
| 72 |
| | |
| 73 |
| +--> per-target TLS check loop (hourly default) |
| 74 |
| | check_tls() --> insert_tls_check() |
| 75 |
| | alert on expiry warning or error |
| 76 |
| | |
| 77 |
| +--> per-peer heartbeat loop (60s default) |
| 78 |
| | GET /api/peer/info --> update mesh state |
| 79 |
| | GET /api/peer/status --> cache for mesh view |
| 80 |
| | grace period state machine on failure |
| 81 |
| | |
| 82 |
| +--> daily prune task (configurable retention) |
| 83 |
| | |
| 84 |
| +--> HTTP API server (Axum, configurable bind address) |
| 85 |
| |
| 86 |
+---> [MCP mode] stdio transport --> tool calls --> same check/query logic |
| 87 |
``` |
| 88 |
|
| 89 |
## Peer Mesh Design |
| 90 |
|
| 91 |
Each PoM instance has a persistent UUID (stored at `~/.local/share/pom/instance_id`). Peers are configured by name with an address, a `on_missing` policy, and an optional grace count. |
| 92 |
|
| 93 |
### Heartbeat State Machine |
| 94 |
|
| 95 |
``` |
| 96 |
Unknown --> (success) --> Online |
| 97 |
Unknown --> (failure) --> GracePeriod --> (failures >= grace_count) --> Missing |
| 98 |
Online --> (failure) --> GracePeriod --> (failures >= grace_count) --> Missing |
| 99 |
Missing --> (success) --> Online (triggers recovery alert) |
| 100 |
``` |
| 101 |
|
| 102 |
Each heartbeat cycle: |
| 103 |
1. GET `/api/peer/info`: verifies identity, measures latency |
| 104 |
2. On first contact, store the peer's UUID in `peer_identities` table |
| 105 |
3. On subsequent contacts, reject UUID mismatches (prevents impersonation) |
| 106 |
4. GET `/api/peer/status`: caches the peer's full status for mesh aggregation |
| 107 |
5. Record heartbeat result in `peer_heartbeats` table |
| 108 |
|
| 109 |
### On Missing Policy |
| 110 |
|
| 111 |
- `alert`: send email alert when peer transitions to Missing, send recovery when it returns |
| 112 |
- `log`: log the event, no email |
| 113 |
- `ignore`: suppress entirely |
| 114 |
|
| 115 |
## Database Schema |
| 116 |
|
| 117 |
SQLite with WAL journal mode. Schema is managed through numbered migrations (currently v1-v13). |
| 118 |
|
| 119 |
### Tables |
| 120 |
|
| 121 |
|
| 122 |
|
| 123 |
| `schema_version` | Migration tracking | version, description, applied_at | |
| 124 |
| `health_checks` | HTTP health check results | target, status, checked_at, response_time_ms, details_json, error | |
| 125 |
| `test_runs` | SSH test execution results | target, started_at, duration_secs, exit_code, passed, summary_json, raw_output | |
| 126 |
| `peer_identities` | First-seen peer UUIDs | peer_name (PK), instance_id, first_seen | |
| 127 |
| `peer_heartbeats` | Heartbeat history | peer_name, status, latency_ms, checked_at | |
| 128 |
| `alerts` | Alert history + cooldown tracking | target, alert_type, from_status, to_status, sent_at | |
| 129 |
| `tls_checks` | TLS certificate probe results | target, host, valid, days_remaining, not_before, not_after, subject, issuer | |
| 130 |
| `incidents` | Health incidents (open/closed) | target, started_at, ended_at, duration_secs, from_status, to_status | |
| 131 |
|
| 132 |
Pre-migration databases are detected by the presence of the `health_checks` table and stamped as v1 without re-running the initial migration. |
| 133 |
|
| 134 |
## Alert Pipeline |
| 135 |
|
| 136 |
``` |
| 137 |
Health status change detected (previous != current) |
| 138 |
| |
| 139 |
+--> operational -> non-operational: send_health_alert(), open incident |
| 140 |
+--> non-operational -> operational: send_health_recovery(), close incidents |
| 141 |
+--> non-operational -> different non-operational: close old incident, open new, alert |
| 142 |
| |
| 143 |
TLS check detects issue |
| 144 |
| |
| 145 |
+--> was OK, now invalid/error: send_tls_error_alert() |
| 146 |
+--> was OK, now within warn_days: send_tls_expiry_alert() |
| 147 |
+--> was bad, now OK: send_tls_recovery() |
| 148 |
| |
| 149 |
Latency drift detected (all recent checks exceed baseline * threshold) |
| 150 |
| |
| 151 |
+--> entered drift: send_latency_drift_alert() |
| 152 |
+--> exited drift: send_latency_recovery() |
| 153 |
| |
| 154 |
Peer transitions to Missing |
| 155 |
| |
| 156 |
+--> send_peer_missing() (if on_missing = alert) |
| 157 |
+--> peer recovers: send_peer_recovery() |
| 158 |
``` |
| 159 |
|
| 160 |
All alerts except recoveries are subject to a per-target cooldown (default 300s). Recoveries always send immediately. Without a Postmark token, alerts are logged to stdout (dev mode). |
| 161 |
|
| 162 |
## API Endpoints |
| 163 |
|
| 164 |
All endpoints require `Authorization: Bearer <token>` when `api_token` is configured (in config or via `POM_API_TOKEN` env var). Without a token configured, all requests pass through. |
| 165 |
|
| 166 |
|
| 167 |
|
| 168 |
| `/api/status` | GET | JSON summary of all targets (latest health, uptime, latency, TLS, staleness, incidents) | |
| 169 |
| `/api/status/{target}` | GET | Same as above for a single target | |
| 170 |
| `/api/trends/{target}` | GET | Latency trend data with configurable window and bucket size (`?hours=24&bucket_minutes=60`) | |
| 171 |
| `/api/versions` | GET | What each target is running: version, git sha, when that version was first seen, when it was last checked, commits behind this host's checkout | |
| 172 |
| `/api/peer/info` | GET | This instance's identity (id, name, version, targets, started_at) | |
| 173 |
| `/api/peer/status` | GET | This instance's full view: identity + target statuses + peer summaries | |
| 174 |
| `/api/mesh` | GET | Aggregated mesh view: self + each peer's cached status | |
| 175 |
|
| 176 |
## Check Types |
| 177 |
|
| 178 |
### HTTP Health Check |
| 179 |
|
| 180 |
Sends GET to the target's health URL. Classifies the response: |
| 181 |
- JSON with `"status": "operational"` --> Operational |
| 182 |
- JSON with `"status": "degraded"` --> Degraded |
| 183 |
- Non-JSON 2xx --> Degraded |
| 184 |
- Non-2xx or unknown status --> Error |
| 185 |
- Connection failure --> Unreachable |
| 186 |
|
| 187 |
Extracts version, git_sha, uptime, checks, and monitoring from the JSON response body. A target that reports `git_sha` gets a commits-behind figure in `pom versions`; one that does not still reports its version. Supports expectation validation: expected status code, required body substrings, and JSON field value assertions (with dot-path traversal for nested fields). |
| 188 |
|
| 189 |
### TLS Certificate Check |
| 190 |
|
| 191 |
Connects to host:port, completes a TLS handshake using the system trust store (webpki-roots), extracts the leaf certificate, and parses it with x509-parser. Records validity, days remaining, not_before/not_after, subject, and issuer. Alerts when days_remaining falls below the configured `warn_days` threshold (default 14). |
| 192 |
|
| 193 |
### SSH Test Runner |
| 194 |
|
| 195 |
Executes a configured command on a remote host via `ssh -o BatchMode=yes`. The command string comes from config (today, `cargo test --workspace` run against a staging checkout). Supports an optional filter argument (validated to `[a-zA-Z0-9_:-]` to prevent injection). Output is parsed for PASS/FAIL step lines and `test result:` cargo test summary lines. |
| 196 |
|
| 197 |
## Key Design Decisions |
| 198 |
|
| 199 |
**SQLite over PostgreSQL.** PoM is a single-binary tool that runs on each monitoring host. SQLite keeps it self-contained with zero external dependencies. WAL mode provides concurrent reads during serve mode. Data volume is modest (a few checks per minute, pruned after 30 days). |
| 200 |
|
| 201 |
**Peer mesh over centralized monitoring.** Two independent instances cross-check each other. If the Hetzner instance goes down, Astra detects it (and vice versa). No single point of failure for the monitoring layer itself. |
| 202 |
|
| 203 |
**Bearer token auth.** Simple, stateless, sufficient for machine-to-machine API access between peers. Configured per-peer and per-instance. No user management needed. |
| 204 |
|
| 205 |
**Versioned migrations.** The migration system detects pre-migration databases and stamps them without re-running. Each migration is a numbered SQL block. This avoids external migration tools while keeping schema evolution safe. |
| 206 |
|
| 207 |
**Separate check intervals.** Health checks can have per-target interval overrides. TLS checks run on a longer interval (hourly default) since certificate state changes slowly. Peer heartbeats run on a short interval (60s default) for timely failure detection. |
| 208 |
|
| 209 |
**Cooldown on alerts.** Prevents alert storms during flapping. Recovery alerts bypass cooldown so operators always know when a service comes back. |
| 210 |
|