Skip to main content

max / makenotwork

synckit: verify + AAD-bind blob content hash, jitter backoff (ultra-fuzz Run #1 Transport) - blob_upload/blob_download take the content hash: upload binds it as AEAD AAD, download decrypts with it and re-verifies sha256(plaintext)==hash, returning IntegrityFailed on mismatch (X1b: substitution/rollback now fails closed). - drop the redundant Bytes::to_vec() copy on download; cap a decrypted blob at 4 GiB to bound memory. - retry backoff gains +/-20% jitter (and the json-retry path now shares retry_delay) so synchronized clients don't retry in lockstep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-06-19 22:29 UTC
Commit: 5ce8d33ce25f9dd67ce6ff5b71cd9a54d7d3461c
Parent: e4266f1
8 files changed, +120 insertions, -39 deletions
M server/docs/pitches/synckit.html +3 -3
@@ -766,13 +766,13 @@ client.<span class="fn">setup_encryption_new</span>(<span class="str">"pass"</sp
766 766 <div class="code-block"><pre><span class="cmt">// Upload: hash locally, request URL, upload encrypted, confirm</span>
767 767 <span class="kw">let</span> resp = client.<span class="fn">blob_upload_url</span>(&amp;hash, size).<span class="kw">await</span>?;
768 768 <span class="kw">if</span> !resp.already_exists {
769 - client.<span class="fn">blob_upload</span>(&amp;resp.upload_url, data).<span class="kw">await</span>?; <span class="cmt">// encrypted on wire</span>
769 + client.<span class="fn">blob_upload</span>(&amp;hash, &amp;resp.upload_url, data).<span class="kw">await</span>?; <span class="cmt">// encrypted + hash-bound on wire</span>
770 770 }
771 771 client.<span class="fn">blob_confirm</span>(&amp;hash, size).<span class="kw">await</span>?;
772 772
773 - <span class="cmt">// Download: request URL, download + decrypt</span>
773 + <span class="cmt">// Download: request URL, download + decrypt + verify content hash</span>
774 774 <span class="kw">let</span> url = client.<span class="fn">blob_download_url</span>(&amp;hash).<span class="kw">await</span>?;
775 - <span class="kw">let</span> plaintext = client.<span class="fn">blob_download</span>(&amp;url).<span class="kw">await</span>?;</pre></div>
775 + <span class="kw">let</span> plaintext = client.<span class="fn">blob_download</span>(&amp;hash, &amp;url).<span class="kw">await</span>?;</pre></div>
776 776
777 777 <div class="highlight-box">
778 778 <p>Blobs are content-addressed by SHA-256 hash. If <code>already_exists</code> returns true, the file is already on the server &mdash; skip the upload. Binary encryption uses raw bytes (no base64 wrapping), so overhead is exactly 40 bytes per file regardless of size.</p>
M shared/synckit-client/Cargo.toml +1
@@ -55,6 +55,7 @@ tracing = "0.1"
55 55
56 56 [dev-dependencies]
57 57 wiremock = "0.6"
58 + sha2 = "0.10"
58 59 # Self dev-dependency: turns on the `testing` feature for this crate's own
59 60 # unit/integration test builds without leaking it into a consumer's default set.
60 61 synckit-client = { path = ".", features = ["testing"] }
M shared/synckit-client/docs/architecture.md +16 -9
@@ -204,15 +204,20 @@ upload:
204 204
205 205 1. `blob_upload_url(hash, size)` requests a presigned S3 PUT URL from the server.
206 206 If the blob already exists (content-addressed by hash), no upload is needed.
207 - 2. `blob_upload(presigned_url, data)` encrypts the plaintext bytes with the
208 - master key and PUTs the ciphertext to S3.
207 + 2. `blob_upload(hash, presigned_url, data)` encrypts the plaintext bytes with the
208 + master key, **binding the content hash as AEAD associated data**, and PUTs the
209 + ciphertext to S3.
209 210 3. `blob_confirm(hash, size)` tells the server the upload completed.
210 211 4. `blob_download_url(hash)` requests a presigned GET URL.
211 - 5. `blob_download(presigned_url)` fetches the ciphertext from S3 and decrypts it.
212 -
213 - Blob encryption uses `encrypt_bytes`/`decrypt_bytes` (raw bytes, no base64) to
214 - avoid the ~33% base64 overhead on large files. Encryption overhead is fixed at
215 - 40 bytes per blob: 24-byte XChaCha20 nonce + 16-byte Poly1305 authentication
212 + 5. `blob_download(expected_hash, presigned_url)` fetches the ciphertext, decrypts
213 + with the hash as AAD, then **re-verifies** that the plaintext hashes to
214 + `expected_hash` (returning `IntegrityFailed` otherwise). The AEAD tag proves
215 + the bytes were not forged; the re-hash proves the server served the bytes for
216 + *this* address, closing the substitution/rollback gap.
217 +
218 + Blob encryption uses `encrypt_bytes_aad`/`decrypt_bytes_aad` (raw bytes, no
219 + base64) to avoid the ~33% base64 overhead on large files. v2 overhead is 44 bytes
220 + per blob: a 4-byte `sk2:` wire tag + 24-byte XChaCha20 nonce + 16-byte Poly1305
216 221 tag.
217 222
218 223 ## Keychain integration
@@ -284,8 +289,10 @@ storing long-lived credentials in memory.
284 289 - **Minimum ciphertext size**: Decryption rejects inputs shorter than 40 bytes
285 290 (24-byte nonce + 16-byte tag), preventing trivial malformed-input attacks.
286 291 - **Ciphertext position binding (AAD)**: v2 entry ciphertext is sealed with
287 - `(table, row_id)` as AEAD associated data, so the untrusted server cannot
288 - relocate a valid ciphertext to another row/table without the open failing
292 + `(table, row_id)` as AEAD associated data, and blob ciphertext with the content
293 + hash; downloaded blobs are additionally re-hashed and rejected if they do not
294 + match their requested address. So the untrusted server cannot relocate or
295 + substitute a valid ciphertext without the open (or the hash check) failing
289 296 closed. See "Wire version tag and AAD binding".
290 297 - **Envelope versioning**: The key envelope includes a version field (`v: 1`)
291 298 and travels with its Argon2 cost parameters, so the work factor can be raised
M shared/synckit-client/src/client/blob.rs +53 -11
@@ -1,15 +1,21 @@
1 1 use bytes::Bytes;
2 + use sha2::{Digest, Sha256};
2 3 use tracing::instrument;
3 4
4 5 use crate::{
5 6 crypto,
6 - error::Result,
7 + error::{Result, SyncKitError},
7 8 types::*,
8 9 };
9 10
10 11 use super::SyncKitClient;
11 12 use super::helpers::check_response;
12 13
14 + /// Upper bound on a single decrypted blob held in memory, guarding against a
15 + /// hostile server returning an absurdly large body that would OOM the client.
16 + /// Generous (4 GiB) so legitimate large media still flow.
17 + const MAX_BLOB_BYTES: usize = 4 * 1024 * 1024 * 1024;
18 +
13 19 impl SyncKitClient {
14 20 /// Request a presigned upload URL for a blob.
15 21 /// Returns (upload_url, already_exists). If `already_exists` is true,
@@ -46,12 +52,15 @@ impl SyncKitClient {
46 52
47 53 /// Upload blob data directly to S3 via a presigned PUT URL.
48 54 ///
49 - /// Encrypts the data with the master key before uploading. The plaintext
50 - /// is never sent to the server, preserving the E2E encryption guarantee.
55 + /// Encrypts the data with the master key before uploading, binding the
56 + /// content `hash` as AEAD associated data so the ciphertext is tied to its
57 + /// content address. The plaintext is never sent to the server, preserving
58 + /// the E2E encryption guarantee.
51 59 #[instrument(skip(self, presigned_url, data))]
52 - pub async fn blob_upload(&self, presigned_url: &str, data: Vec<u8>) -> Result<()> {
60 + pub async fn blob_upload(&self, hash: &str, presigned_url: &str, data: Vec<u8>) -> Result<()> {
53 61 let master_key = self.require_master_key()?;
54 - let encrypted = Bytes::from(crypto::encrypt_bytes(&data, &master_key)?);
62 + let encrypted =
63 + Bytes::from(crypto::encrypt_bytes_aad(&data, &master_key, hash.as_bytes())?);
55 64
56 65 self.retry_request(|| {
57 66 let req = self
@@ -118,12 +127,19 @@ impl SyncKitClient {
118 127 Ok(download.download_url)
119 128 }
120 129
121 - /// Download blob data from S3 via a presigned GET URL.
130 + /// Download blob data from S3 via a presigned GET URL and verify it.
131 + ///
132 + /// Decrypts with the master key (binding the content `expected_hash` as AEAD
133 + /// associated data), then **re-verifies** that the plaintext hashes to
134 + /// `expected_hash`. The AEAD tag alone proves the bytes were not forged, but
135 + /// not that the untrusted server served the bytes for *this* address; the
136 + /// re-hash closes the substitution/rollback gap. Returns
137 + /// [`SyncKitError::IntegrityFailed`] on mismatch.
122 138 ///
123 - /// Decrypts the data with the master key after downloading. The server
124 - /// only ever stores encrypted blobs, preserving the E2E encryption guarantee.
139 + /// The blob is held in memory (the caller writes it to disk); the decrypted
140 + /// size is capped at [`MAX_BLOB_BYTES`].
125 141 #[instrument(skip(self, presigned_url))]
126 - pub async fn blob_download(&self, presigned_url: &str) -> Result<Vec<u8>> {
142 + pub async fn blob_download(&self, expected_hash: &str, presigned_url: &str) -> Result<Vec<u8>> {
127 143 let resp = self
128 144 .retry_request(|| {
129 145 let req = self.http.get(presigned_url);
@@ -131,9 +147,24 @@ impl SyncKitClient {
131 147 })
132 148 .await?;
133 149
134 - let encrypted = resp.bytes().await?.to_vec();
150 + // Owned `Bytes`; decrypt borrows it directly (no redundant copy).
151 + let encrypted = resp.bytes().await?;
152 + if encrypted.len() > MAX_BLOB_BYTES {
153 + return Err(SyncKitError::Internal(format!(
154 + "blob exceeds {MAX_BLOB_BYTES}-byte limit",
155 + )));
156 + }
135 157 let master_key = self.require_master_key()?;
136 - crypto::decrypt_bytes(&encrypted, &master_key)
158 + let plaintext = crypto::decrypt_bytes_aad(&encrypted, &master_key, expected_hash.as_bytes())?;
159 +
160 + let actual = format!("{:x}", Sha256::digest(&plaintext));
161 + if actual != expected_hash {
162 + return Err(SyncKitError::IntegrityFailed {
163 + expected: expected_hash.to_string(),
164 + actual,
165 + });
166 + }
167 + Ok(plaintext)
137 168 }
138 169 }
139 170
@@ -167,6 +198,17 @@ mod tests {
167 198 }
168 199
169 200 #[test]
201 + fn blob_content_hash_format_matches_consumer() {
202 + use sha2::{Digest, Sha256};
203 + // The integrity check in blob_download compares against this exact form:
204 + // lowercase hex of SHA-256, the same string consumers store as the blob
205 + // hash. If this drifts, every verified download would falsely reject.
206 + let h = format!("{:x}", Sha256::digest(b"hello blob"));
207 + assert_eq!(h.len(), 64);
208 + assert!(h.chars().all(|c| c.is_ascii_hexdigit() && !c.is_ascii_uppercase()));
209 + }
210 +
211 + #[test]
170 212 fn blob_confirm_request_serialization() {
171 213 let req = BlobConfirmRequest {
172 214 hash: "sha256-def456".to_string(),
M shared/synckit-client/src/client/helpers.rs +18 -3
@@ -69,7 +69,7 @@ impl SyncKitClient {
69 69 Err(e) => {
70 70 let err = SyncKitError::Http(e);
71 71 if attempt < MAX_RETRIES {
72 - let delay = BASE_DELAY * 2u32.pow(attempt);
72 + let delay = retry_delay(&err, attempt);
73 73 tracing::debug!(
74 74 attempt = attempt + 1,
75 75 max_retries = MAX_RETRIES,
@@ -339,13 +339,28 @@ fn parse_retry_after(resp: &reqwest::Response) -> Option<u64> {
339 339 /// Compute the backoff delay for a retry attempt.
340 340 ///
341 341 /// Uses the server's `Retry-After` header (if present on a 429) capped at
342 - /// 60 seconds. Falls back to exponential backoff (1s, 2s, 4s).
342 + /// 60 seconds. Otherwise exponential backoff (1s, 2s, 4s) with ±20% jitter so
343 + /// many clients that hit a transient blip simultaneously don't retry in lockstep
344 + /// and re-amplify the load.
343 345 fn retry_delay(err: &SyncKitError, attempt: u32) -> std::time::Duration {
344 346 if let SyncKitError::Server { retry_after_secs: Some(secs), .. } = err {
345 347 let capped = (*secs).min(60);
346 348 return std::time::Duration::from_secs(capped);
347 349 }
348 - BASE_DELAY * 2u32.pow(attempt)
350 + jittered(BASE_DELAY * 2u32.pow(attempt))
351 + }
352 +
353 + /// Spread a backoff delay by ±20% to avoid synchronized retries (thundering herd).
354 + fn jittered(base: std::time::Duration) -> std::time::Duration {
355 + use rand::Rng;
356 + let millis = base.as_millis() as u64;
357 + let span = millis / 5; // 20%
358 + if span == 0 {
359 + return base;
360 + }
361 + // Uniform in [millis - span, millis + span].
362 + let delta = rand::rng().random_range(0..=2 * span);
363 + std::time::Duration::from_millis(millis - span + delta)
349 364 }
350 365
351 366 /// Returns true if the error is transient and worth retrying.
M shared/synckit-client/src/crypto.rs +4
@@ -397,6 +397,10 @@ pub fn decrypt_bytes_aad(
397 397 /// Encryption overhead in bytes (24-byte nonce + 16-byte Poly1305 tag).
398 398 pub const ENCRYPTION_OVERHEAD: usize = NONCE_SIZE + 16;
399 399
400 + /// Overhead of a v2 (AAD-bound, `sk2:`-tagged) blob: the wire tag plus the
401 + /// nonce + Poly1305 tag.
402 + pub const ENCRYPTION_OVERHEAD_V2: usize = WIRE_V2_TAG_BYTES.len() + ENCRYPTION_OVERHEAD;
403 +
400 404 /// Encrypt a JSON value (legacy, untagged, empty AAD). Prefer [`encrypt_json_aad`].
401 405 pub fn encrypt_json(
402 406 value: &serde_json::Value,
M shared/synckit-client/src/error.rs +6
@@ -35,6 +35,12 @@ pub enum SyncKitError {
35 35 #[error("Wrong password or corrupted key envelope")]
36 36 DecryptionFailed,
37 37
38 + /// A downloaded blob decrypted cleanly but did not hash to its requested
39 + /// content address — the server served a different-but-authentic or
40 + /// rolled-back ciphertext under this hash. The bytes are discarded.
41 + #[error("Blob integrity check failed: expected {expected}, got {actual}")]
42 + IntegrityFailed { expected: String, actual: String },
43 +
38 44 /// Key envelope JSON has an unrecognized version or missing fields.
39 45 #[error("Invalid key envelope: {0}")]
40 46 InvalidEnvelope(String),
M shared/synckit-client/tests/integration.rs +19 -13
@@ -8,6 +8,7 @@ use std::sync::Arc;
8 8 use base64::Engine;
9 9 use chrono::Utc;
10 10 use serde_json::json;
11 + use sha2::Digest;
11 12 use uuid::Uuid;
12 13 use wiremock::matchers::{method, path};
13 14 use wiremock::{Mock, MockServer, ResponseTemplate};
@@ -443,7 +444,7 @@ async fn blob_upload_encrypts_data() {
443 444 let plaintext = b"hello blob data";
444 445 let presigned = format!("{}{}", server.uri(), upload_path);
445 446 client
446 - .blob_upload(&presigned, plaintext.to_vec())
447 + .blob_upload("sha256-test", &presigned, plaintext.to_vec())
447 448 .await
448 449 .unwrap();
449 450
@@ -472,8 +473,10 @@ async fn blob_download_decrypts_data() {
472 473 let key = synckit_client::crypto::generate_master_key();
473 474 client.set_master_key_raw(key);
474 475
475 - // Encrypt data to simulate what S3 would return
476 + // Encrypt data to simulate what S3 would return. A legacy (untagged) blob
477 + // still decrypts through the AAD-aware reader and must pass the hash check.
476 478 let plaintext = b"decrypted blob content";
479 + let hash = format!("{:x}", sha2::Sha256::digest(plaintext));
477 480 let encrypted = synckit_client::crypto::encrypt_bytes(plaintext, &key).unwrap();
478 481
479 482 let download_path = "/s3/download";
@@ -484,7 +487,7 @@ async fn blob_download_decrypts_data() {
484 487 .await;
485 488
486 489 let presigned = format!("{}{}", server.uri(), download_path);
487 - let result = client.blob_download(&presigned).await.unwrap();
490 + let result = client.blob_download(&hash, &presigned).await.unwrap();
488 491 assert_eq!(result, plaintext);
489 492 }
490 493
@@ -511,7 +514,7 @@ async fn blob_upload_retries_on_503() {
511 514 client.set_master_key_raw(key);
512 515
513 516 let presigned = format!("{}{}", server.uri(), upload_path);
514 - let result = client.blob_upload(&presigned, b"data".to_vec()).await;
517 + let result = client.blob_upload("sha256-x", &presigned, b"data".to_vec()).await;
515 518 assert!(result.is_ok(), "Should succeed after retry: {result:?}");
516 519 }
517 520
@@ -1301,10 +1304,10 @@ async fn blob_upload_zero_byte_data() {
1301 1304 client.set_master_key_raw(key);
1302 1305
1303 1306 let presigned = format!("{}{}", server.uri(), upload_path);
1304 - let result = client.blob_upload(&presigned, vec![]).await;
1307 + let result = client.blob_upload("sha256-empty", &presigned, vec![]).await;
1305 1308 assert!(result.is_ok(), "Zero-byte blob upload should succeed");
1306 1309
1307 - // Verify the uploaded data has encryption overhead only
1310 + // Verify the uploaded data has encryption overhead only (v2: + sk2: tag)
1308 1311 let requests = server.received_requests().await.unwrap();
1309 1312 let req = requests
1310 1313 .iter()
@@ -1312,7 +1315,7 @@ async fn blob_upload_zero_byte_data() {
1312 1315 .unwrap();
1313 1316 assert_eq!(
1314 1317 req.body.len(),
1315 - synckit_client::crypto::ENCRYPTION_OVERHEAD,
1318 + synckit_client::crypto::ENCRYPTION_OVERHEAD_V2,
1316 1319 "Empty plaintext should produce exactly overhead bytes"
1317 1320 );
1318 1321 }
@@ -1326,6 +1329,7 @@ async fn blob_upload_download_roundtrip() {
1326 1329 client.set_master_key_raw(key);
1327 1330
1328 1331 let plaintext = b"roundtrip blob data with special bytes \x00\xFF\x01";
1332 + let hash = format!("{:x}", sha2::Sha256::digest(plaintext));
1329 1333
1330 1334 // Upload
1331 1335 let upload_path = "/s3/roundtrip-upload";
@@ -1337,6 +1341,7 @@ async fn blob_upload_download_roundtrip() {
1337 1341
1338 1342 client
1339 1343 .blob_upload(
1344 + &hash,
1340 1345 &format!("{}{}", server.uri(), upload_path),
1341 1346 plaintext.to_vec(),
1342 1347 )
@@ -1360,7 +1365,7 @@ async fn blob_upload_download_roundtrip() {
1360 1365 .await;
1361 1366
1362 1367 let downloaded = client
1363 - .blob_download(&format!("{}{}", server.uri(), download_path))
1368 + .blob_download(&hash, &format!("{}{}", server.uri(), download_path))
1364 1369 .await
1365 1370 .unwrap();
1366 1371
@@ -1907,7 +1912,7 @@ async fn blob_download_with_wrong_key_fails() {
1907 1912 client.set_master_key_raw(key2);
1908 1913
1909 1914 let result = client
1910 - .blob_download(&format!("{}{}", server.uri(), download_path))
1915 + .blob_download("sha256-x", &format!("{}{}", server.uri(), download_path))
1911 1916 .await;
1912 1917
1913 1918 assert!(
@@ -2461,6 +2466,7 @@ async fn blob_download_retries_on_503() {
2461 2466 client.set_master_key_raw(key);
2462 2467
2463 2468 let plaintext = b"retry download test";
2469 + let hash = format!("{:x}", sha2::Sha256::digest(plaintext));
2464 2470 let encrypted = synckit_client::crypto::encrypt_bytes(plaintext, &key).unwrap();
2465 2471
2466 2472 let download_path = "/s3/retry-download";
@@ -2478,7 +2484,7 @@ async fn blob_download_retries_on_503() {
2478 2484 .await;
2479 2485
2480 2486 let presigned = format!("{}{}", server.uri(), download_path);
2481 - let result = client.blob_download(&presigned).await.unwrap();
2487 + let result = client.blob_download(&hash, &presigned).await.unwrap();
2482 2488 assert_eq!(result, plaintext);
2483 2489 }
2484 2490
@@ -2499,14 +2505,14 @@ async fn blob_upload_1mb_with_correct_overhead() {
2499 2505
2500 2506 let plaintext: Vec<u8> = (0..1_048_576u32).map(|i| (i % 256) as u8).collect();
2501 2507 let presigned = format!("{}{}", server.uri(), upload_path);
2502 - client.blob_upload(&presigned, plaintext.clone()).await.unwrap();
2508 + client.blob_upload("sha256-1mb", &presigned, plaintext.clone()).await.unwrap();
2503 2509
2504 2510 let requests = server.received_requests().await.unwrap();
2505 2511 let upload_req = requests.iter().find(|r| r.url.path() == upload_path).unwrap();
2506 2512 assert_eq!(
2507 2513 upload_req.body.len(),
2508 - plaintext.len() + synckit_client::crypto::ENCRYPTION_OVERHEAD,
2509 - "1MB upload should have exactly ENCRYPTION_OVERHEAD bytes added"
2514 + plaintext.len() + synckit_client::crypto::ENCRYPTION_OVERHEAD_V2,
2515 + "1MB upload should have exactly ENCRYPTION_OVERHEAD_V2 bytes added"
2510 2516 );
2511 2517 }
2512 2518