max / makenotwork

sando: document the rollback contract + gate migration-bearing promotes Rollback restores the binary and release_contents only; the database migrates forward on boot and sqlx has no down path, so a promote that carries a migration is a one-way door. Nothing said so, and nothing made the operator pause. - Document the rollback contract in deploy/README.md: forward is safe, back serves a mismatched schema; undoing a migration needs a manual DB restore; migration_dry_run already restore-tests the latest backup each build, but a full restore-to-serving drill is not automated (infra follow-up). - Add a bears_migration flag to the promote request. When set, the predecessor tier must show a fresh manual_confirm before the advance, even if it configures none; hotfix (which skips only burn_in) does not suppress it. Reuses the existing manual_confirm freshness machinery by injecting the gate. Auto-detecting migrations (via a migration_dry_run applied-count) is a possible future hardening; migration_dry_run records no such count today. Build, clippy -D warnings, and the sando suite (243 tests, +1 new) stay green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-07-25 01:43 UTC

Commit: 14eacfbf75312c3e02fcbb7e9b2ce931f2b97b00

Parent: 5471e72

3 files changed, +118 insertions, -1 deletion

M sando/daemon/src/routes/mod.rs +71

			@@ -246,6 +246,15 @@ struct PromoteBody {
246	246		hotfix: bool,
247	247		#[serde(default)]
248	248		reset_burn_in: bool,
	249	+	/// Set when the release carries a database migration. Rollback restores the
	250	+	/// binary and release_contents only, never the schema (MNW migrates forward
	251	+	/// on boot and has no down path), so a migration-bearing promote is one-way.
	252	+	/// This forces a fresh `manual_confirm` on the predecessor tier even if the
	253	+	/// tier does not configure one, so the operator consciously acknowledges the
	254	+	/// rollback contract. `hotfix` does not suppress it. See
	255	+	/// `deploy/README.md` "Rollback contract".
	256	+	#[serde(default)]
	257	+	bears_migration: bool,
249	258		}
250	259
251	260		async fn promote(
			@@ -1578,6 +1587,68 @@ mod tests {
1578	1587		}
1579	1588
1580	1589		#[tokio::test]
	1590	+	async fn migration_bearing_promote_requires_fresh_confirm() {
	1591	+	// The predecessor (host) configures NO gates, so nothing would normally
	1592	+	// block host -> a. A `bears_migration` promote must still be blocked on a
	1593	+	// fresh `manual_confirm` it does not have: rollback restores the binary
	1594	+	// only, so the one-way advance needs a conscious operator sign-off.
	1595	+	let pool = fresh_pool().await;
	1596	+	for (i, name) in ["host", "a"].iter().enumerate() {
	1597	+	sqlx::query(
	1598	+	"INSERT INTO tiers (name, ord, provisioned, canary) VALUES (?, ?, 1, 'sequential')",
	1599	+	)
	1600	+	.bind(name)
	1601	+	.bind(i as i64)
	1602	+	.execute(&pool)
	1603	+	.await
	1604	+	.unwrap();
	1605	+	sqlx::query("INSERT INTO tier_state (tier) VALUES (?)")
	1606	+	.bind(name)
	1607	+	.execute(&pool)
	1608	+	.await
	1609	+	.unwrap();
	1610	+	}
	1611	+	let mut topo = test_topo();
	1612	+	topo.tiers[0].gates = vec![]; // host configures no gates at all
	1613	+	let executors = Arc::new(crate::state::build_executors(&topo));
	1614	+	let state = AppState {
	1615	+	pool,
	1616	+	topo: Arc::new(topo),
	1617	+	cfg: Arc::new(test_cfg()),
	1618	+	active_build: Arc::new(tokio::sync::Mutex::new(None)),
	1619	+	deploy_lock: Arc::new(tokio::sync::Mutex::new(())),
	1620	+	events: crate::events::channel(),
	1621	+	executors,
	1622	+	api_token: None,
	1623	+	};
	1624	+	sqlx::query("INSERT INTO versions (version, git_sha, built_at, artifact_path) VALUES ('0.8.12','sha',datetime('now'),'/tmp/x')")
	1625	+	.execute(&state.pool).await.unwrap();
	1626	+	sqlx::query("UPDATE tier_state SET current_version = '0.8.12' WHERE tier = 'host'")
	1627	+	.execute(&state.pool)
	1628	+	.await
	1629	+	.unwrap();
	1630	+
	1631	+	let app = router(state);
	1632	+	let resp = app
	1633	+	.oneshot(
	1634	+	Request::builder()
	1635	+	.method("POST")
	1636	+	.uri("/promote/a")
	1637	+	.header("content-type", "application/json")
	1638	+	.body(Body::from(r#"{"bears_migration": true}"#))
	1639	+	.unwrap(),
	1640	+	)
	1641	+	.await
	1642	+	.unwrap();
	1643	+	assert_eq!(resp.status(), StatusCode::CONFLICT);
	1644	+	let body = body_string(resp).await;
	1645	+	assert!(
	1646	+	body.contains("manual_confirm"),
	1647	+	"expected the migration promote to block on manual_confirm; got: {body}"
	1648	+	);
	1649	+	}
	1650	+
	1651	+	#[tokio::test]
1581	1652		async fn tier_state_advance_is_atomic_previous_from_old_current() {
1582	1653		// CF3: the promote advance is a single UPDATE where previous_version is
1583	1654		// set from the row's old current_version (SQLite evaluates RHS against

M sando/daemon/src/routes/promotion.rs +16 -1

			@@ -132,10 +132,25 @@ pub(super) async fn promote_inner(
132	132		// build (with optional hotfix override that skips burn_in). Evaluated
133	133		// against the topology gate list, so a gate that never ran blocks the
134	134		// promote instead of being treated as green.
	135	+	//
	136	+	// A migration-bearing promote (`bears_migration`) forces a fresh
	137	+	// `manual_confirm` on the predecessor even if the tier does not configure
	138	+	// one: rollback restores the binary + release_contents only, never the
	139	+	// schema, so the operator must consciously acknowledge the one-way advance
	140	+	// (deploy/README.md "Rollback contract"). `hotfix` skips only `burn_in`, so
	141	+	// it does not suppress this confirm.
	142	+	let mut effective_gates = source.gates.clone();
	143	+	if body.bears_migration
	144	+	&& !effective_gates
	145	+	.iter()
	146	+	.any(\|g\| matches!(g, crate::topology::Gate::ManualConfirm))
	147	+	{
	148	+	effective_gates.push(crate::topology::Gate::ManualConfirm);
	149	+	}
135	150		let pending = unsatisfied_gates(
136	151		&s.pool,
137	152		&source.name,
138		-	&source.gates,
	153	+	&effective_gates,
139	154		&version_str,
140	155		build_id,
141	156		body.hotfix,

M sando/deploy/README.md +31

			@@ -79,6 +79,37 @@ curl -sS -X POST "$BASE/self-update" -H 'Content-Type: application/json' \
79	79		journalctl -u "sando-update@$SHA" -f
80	80		```
81	81
	82	+	## Rollback contract
	83	+
	84	+	A Sando rollback (canary rollback of a node, or an operator swapping the
	85	+	`current` symlink back to an older release dir) restores **the binary and the
	86	+	release contents only**. The database does not roll back:
	87	+
	88	+	- MNW migrates forward on boot (`makenotwork.service` runs pending migrations
	89	+	when it starts).
	90	+	- `sqlx::migrate::Migrator` has no down path, and Sando never invokes one.
	91	+
	92	+	So rolling, say, `0.10.15` back to `0.10.14` after `0.10.15` applied a
	93	+	migration leaves the old binary running against a newer schema. That is a
	94	+	one-way door for any release that carries a migration: forward is safe, back is
	95	+	not. Sando has no restore-to-prod path either — `/backup/fetch` pulls the prod
	96	+	dump solely as input to the `migration_dry_run` gate, not to restore a live node.
	97	+
	98	+	Operating rule. Promote a migration-bearing release with
	99	+	`{"bears_migration": true}`. That forces a fresh `manual_confirm` on the
	100	+	predecessor tier before the advance, even on a tier that configures no confirm,
	101	+	so the one-way advance is a conscious step. `hotfix` does not suppress it (it
	102	+	skips only `burn_in`). To actually undo a migration-bearing release you must
	103	+	restore the database from a backup by hand first; a symlink rollback alone will
	104	+	serve a mismatched schema.
	105	+
	106	+	Does the backup actually restore? The `migration_dry_run` gate answers this
	107	+	on every build: it resets a scratch database, restores the latest
	108	+	`/srv/sando/backups/latest.sql.gz`, and runs the migrator against it. A failed
	109	+	restore fails the gate. What is not automated is a full restore-to-serving
	110	+	drill (restore into a throwaway target and confirm the app boots and serves
	111	+	against it) — that is tracked as an infra task, not wired into the pipeline.
	112	+
82	113		## Companion services (deploying mnw-cli in lockstep)
83	114
84	115		`mnw-cli` (the public git-SSH server that proxies to `/api/internal/*`) shares