| 1 |
# SyncKit Client SDK, Architecture |
| 2 |
|
| 3 |
## Overview |
| 4 |
|
| 5 |
The SyncKit Client SDK (`synckit-client`) is a Rust crate that provides |
| 6 |
end-to-end encrypted cloud sync against the MNW SyncKit server. Consumer apps |
| 7 |
(GoingsOn, Balanced Breakfast, audiofiles) use this crate to push and pull |
| 8 |
changelog entries without the server ever seeing plaintext data. |
| 9 |
|
| 10 |
Version: 0.5.0. |
| 11 |
|
| 12 |
## Crate structure |
| 13 |
|
| 14 |
``` |
| 15 |
src/ |
| 16 |
lib.rs Crate root. Re-exports public types, quick-start doc example. |
| 17 |
client/ SyncKitClient struct and all HTTP methods, split across |
| 18 |
submodules (auth, sync, blob, encryption, rotation, |
| 19 |
subscribe, subscription, helpers). Retry logic and token |
| 20 |
expiry detection live here. |
| 21 |
crypto.rs Encryption engine. Key derivation (Argon2id), key wrapping |
| 22 |
(XChaCha20-Poly1305), per-entry encrypt/decrypt, blob |
| 23 |
encrypt/decrypt, ZeroizeOnDrop guard. |
| 24 |
error.rs SyncKitError enum (thiserror, #[non_exhaustive]): Http, Server, |
| 25 |
Json, NoMasterKey, DecryptionFailed, IntegrityFailed, |
| 26 |
InvalidEnvelope, Crypto, Base64, NotAuthenticated, TokenExpired, |
| 27 |
InvalidArgument, Internal, and Keychain (feature-gated behind |
| 28 |
`keychain`). |
| 29 |
keystore.rs OS keychain integration (macOS Keychain, Linux secret-service, |
| 30 |
Windows Credential Manager) via the `keyring` crate. |
| 31 |
Feature-gated behind `keychain` (default on). No-op stubs |
| 32 |
when disabled. |
| 33 |
types.rs Request/response types matching the server wire protocol. |
| 34 |
Public types: ChangeEntry, ChangeOp, Device, SyncStatus, |
| 35 |
BlobUploadUrlResponse. Internal types: auth, push/pull, key, |
| 36 |
OAuth, blob requests. |
| 37 |
|
| 38 |
Public re-exports from lib.rs: |
| 39 |
SyncKitClient, SyncKitConfig, SessionInfo, |
| 40 |
ChangeEntry, ChangeOp, Device, SyncStatus, |
| 41 |
SyncKitError, Result |
| 42 |
``` |
| 43 |
|
| 44 |
## Key lifecycle |
| 45 |
|
| 46 |
``` |
| 47 |
password |
| 48 |
| |
| 49 |
v |
| 50 |
Argon2id (64 MB, 3 iterations, parallelism 1) |
| 51 |
+ random 32-byte salt (stored in envelope) |
| 52 |
| |
| 53 |
v |
| 54 |
wrapping_key (256-bit) |
| 55 |
| |
| 56 |
v |
| 57 |
XChaCha20-Poly1305 encrypt/decrypt |
| 58 |
| |
| 59 |
v |
| 60 |
master_key (256-bit, randomly generated on first device) |
| 61 |
| |
| 62 |
v |
| 63 |
XChaCha20-Poly1305 per-entry / per-blob encrypt/decrypt |
| 64 |
``` |
| 65 |
|
| 66 |
- **First device**: generates a random master key, wraps it with the password, |
| 67 |
pushes the encrypted envelope to the server, caches plaintext in OS keychain. |
| 68 |
- **Subsequent devices**: pulls the encrypted envelope from the server, unwraps |
| 69 |
with the password, caches in OS keychain. |
| 70 |
- **Subsequent launches**: loads the master key directly from the OS keychain |
| 71 |
(no password prompt, no server call). |
| 72 |
- **Password change**: decrypts master key with old password, re-wraps with new |
| 73 |
password (fresh random salt), pushes new envelope to server. The master key |
| 74 |
itself does not change, so existing ciphertext remains valid. |
| 75 |
|
| 76 |
## Authentication flow |
| 77 |
|
| 78 |
### Email/password |
| 79 |
|
| 80 |
1. `authenticate(email, password)` POSTs to `/api/sync/auth` with the app API key. |
| 81 |
2. Server validates credentials and returns a JWT, user ID, and app ID. |
| 82 |
3. Client stores the session in an internal `Mutex<Option<Session>>`. |
| 83 |
|
| 84 |
### OAuth2 PKCE |
| 85 |
|
| 86 |
1. Caller generates a PKCE code verifier and challenge. |
| 87 |
2. `build_authorize_url()` constructs the `/oauth/authorize` URL with the challenge. |
| 88 |
3. Caller opens the URL in a browser and starts a localhost callback server. |
| 89 |
4. After user authorizes, `authenticate_with_code(code, verifier, port)` exchanges |
| 90 |
the authorization code for a JWT via `/oauth/token`. |
| 91 |
5. Session is stored identically to email/password auth. |
| 92 |
|
| 93 |
### Session restoration |
| 94 |
|
| 95 |
`restore_session(token, user_id, app_id)` populates the session from stored |
| 96 |
credentials (e.g., OS keychain) without making any HTTP calls. The caller is |
| 97 |
responsible for checking `is_token_expired()` and re-authenticating if needed. |
| 98 |
|
| 99 |
### Session utilities |
| 100 |
|
| 101 |
- `session_info()` returns `Option<SessionInfo>` (token, user_id, app_id) |
| 102 |
without making HTTP calls. Returns `None` if not authenticated. |
| 103 |
- `is_token_expired()` checks the JWT `exp` claim with a 30-second buffer. |
| 104 |
Returns `true` if no session exists or the token is about to expire. |
| 105 |
- `clear_session()` clears in-memory session and master key. Does not |
| 106 |
affect OS keychain (call `keystore::delete_key` separately). |
| 107 |
- `config()` returns a reference to the `SyncKitConfig`. |
| 108 |
|
| 109 |
## Encryption setup flow |
| 110 |
|
| 111 |
``` |
| 112 |
has_server_key()? |
| 113 |
| |
| 114 |
+-- false --> setup_encryption_new(password) |
| 115 |
| Generate master key, wrap, push envelope, cache in keychain |
| 116 |
| |
| 117 |
+-- true --> setup_encryption_existing(password) |
| 118 |
Pull envelope, unwrap with password, cache in keychain |
| 119 |
``` |
| 120 |
|
| 121 |
On subsequent launches, `try_load_key_from_keychain()` loads the master key |
| 122 |
from the OS keychain without any server interaction or password prompt. |
| 123 |
|
| 124 |
## Push/Pull protocol |
| 125 |
|
| 126 |
### The HLC envelope |
| 127 |
|
| 128 |
Every change, **including Deletes**, is encrypted as a JSON *envelope*, not as |
| 129 |
the bare row data: |
| 130 |
|
| 131 |
```json |
| 132 |
{ "__skver": 2, "__skhlc": { "wall_ms": …, "counter": …, "node": "…" }, "data": <row|null> } |
| 133 |
``` |
| 134 |
|
| 135 |
Encrypting the envelope (rather than just the row) is what carries the hybrid |
| 136 |
logical clock (HLC) inside the E2E ciphertext, so the server can order entries by |
| 137 |
`seq` but never sees or orders by the clock. A Delete has `data: null` but still |
| 138 |
seals an envelope, so its HLC travels too. On decrypt, a payload whose embedded |
| 139 |
`__skhlc` parses as an `Hlc` is treated as an envelope; anything else is a legacy |
| 140 |
bare row whose HLC is synthesized from `node` + the entry's `client_timestamp`. |
| 141 |
|
| 142 |
### Wire version tag and AAD binding |
| 143 |
|
| 144 |
The encrypted `data` string is **position-bound**. v2 ciphertext is prefixed |
| 145 |
`sk2:` and sealed with AEAD associated data `aad_for_entry(table, row_id)` = |
| 146 |
`table` `0x1f` `row_id`. Because the tag is read before decryption, the reader |
| 147 |
always knows which AAD to apply: |
| 148 |
|
| 149 |
- `sk2:`-tagged → open with the entry's `(table, row_id)` AAD. |
| 150 |
- untagged (legacy, pre-v2) → open with empty AAD. |
| 151 |
|
| 152 |
This means a malicious or compromised server cannot relocate a valid ciphertext |
| 153 |
to a different row or table: the AAD no longer matches and the open fails closed. |
| 154 |
Legacy and v2 ciphertext coexist with no flag-day; key rotation re-encrypts |
| 155 |
legacy entries into the v2 form opportunistically. |
| 156 |
|
| 157 |
The AAD is never passed as raw bytes. Callers name a context, `AeadContext::Entry |
| 158 |
{ table, row_id }` or `AeadContext::Blob { hash }`, and the crypto layer derives |
| 159 |
the AAD from it. The byte-level builder is private, so an empty or mismatched AAD |
| 160 |
is not expressible at the call site. |
| 161 |
|
| 162 |
### Envelope version dispatch (`__skver`) |
| 163 |
|
| 164 |
The decrypted entry payload is an envelope `{ __skver: 2, __skhlc, data }`. |
| 165 |
Splitting it back into `(hlc, data)` dispatches on the explicit `__skver` tag via |
| 166 |
a typed `WireVersion`: an unknown future version is a hard error, not a silent |
| 167 |
fall-through to a bare-row read. A tag-less payload carrying an embedded `__skhlc` |
| 168 |
is a gen-1 envelope; anything else is a legacy bare row. |
| 169 |
|
| 170 |
### Chunked blob format (`sk3:`) |
| 171 |
|
| 172 |
Blobs use a chunked AEAD format so a reader decrypts and verifies one chunk at a |
| 173 |
time. Layout: `sk3:` `version(u8=3)` `chunk_size(u32 LE)` `total_len(u64 LE)` |
| 174 |
followed by sealed 1 MiB chunks. Each chunk's AAD binds `(content_hash, |
| 175 |
chunk_index, chunk_count)`, so reorder, truncation, duplication, and cross-blob |
| 176 |
substitution all fail closed. `blob_download` streams the body and decrypts |
| 177 |
chunk-by-chunk (peak memory is the plaintext plus one in-flight chunk, not |
| 178 |
ciphertext *and* plaintext together); legacy `sk2:`/untagged blobs still decrypt |
| 179 |
whole-buffer with no flag-day. |
| 180 |
|
| 181 |
### Push |
| 182 |
|
| 183 |
1. Caller provides a `Vec<ChangeEntry>` with plaintext `data` fields and an HLC. |
| 184 |
2. Client wraps each `(hlc, data)` in an envelope, encrypts it with the master |
| 185 |
key (XChaCha20-Poly1305, random nonce, `(table, row_id)` AAD), and emits the |
| 186 |
`sk2:`-tagged base64 string. |
| 187 |
3. The encrypted payload is POSTed to `/api/sync/push` with the device ID and a |
| 188 |
client-generated `batch_id` (idempotent push: a replayed batch returns the |
| 189 |
existing cursor). |
| 190 |
4. Server appends entries to the changelog and returns the new cursor (i64). |
| 191 |
5. Retries on transient failures (see below). |
| 192 |
|
| 193 |
### Pull |
| 194 |
|
| 195 |
1. Client POSTs to `/api/sync/pull` with the device ID and last-known cursor. |
| 196 |
2. Server returns changelog entries since that cursor, a new cursor, and a |
| 197 |
`has_more` flag for pagination. |
| 198 |
3. Client decrypts each entry's envelope (auto-detecting the wire version) and |
| 199 |
returns plaintext `ChangeEntry` values, HLC included, to the caller. |
| 200 |
4. Retries on transient failures. |
| 201 |
|
| 202 |
**Caller contract.** The SDK does not persist the cursor or dedup re-delivered |
| 203 |
changes. Persist the returned cursor only *after* applying the returned changes |
| 204 |
(same transaction where possible), and make apply idempotent per |
| 205 |
`(table, row_id, hlc)`, a reconnect can re-deliver a batch. |
| 206 |
|
| 207 |
### Conflict resolution |
| 208 |
|
| 209 |
Conflicts are resolved client-side (the server cannot read plaintext). |
| 210 |
`resolve_lww` orders by HLC, operation-agnostically: the higher `(wall_ms, |
| 211 |
counter, node)` wins, so a newer edit beats an older delete and vice-versa, and |
| 212 |
every device converges on the same winner. An *exactly* equal HLC normally means |
| 213 |
a same-device echo; if two installs share a `node` UUID (cloned config), the tie |
| 214 |
is broken deterministically on the canonical payload bytes so both devices still |
| 215 |
converge. |
| 216 |
|
| 217 |
`detect_conflicts` only flags rows with an *un-pushed* local edit. Its clean set |
| 218 |
is returned as an opaque `CleanChanges`, not a bare `Vec`: the only way to read |
| 219 |
the applicable entries out is `gated(committed_hlc)`, which drops any clean change |
| 220 |
older-or-equal to the row's committed HLC. (`row_keys()` lets the caller pre-fetch |
| 221 |
those clocks first.) This makes "apply a clean change without checking the |
| 222 |
committed clock", which would let an older remote edit clobber a newer local |
| 223 |
value, unrepresentable rather than a contract the caller has to remember. |
| 224 |
|
| 225 |
### Sequence numbers |
| 226 |
|
| 227 |
The server assigns a monotonically increasing sequence number (`seq`) to each |
| 228 |
changelog entry. The cursor is the `seq` of the last entry the client has seen. |
| 229 |
Pulling with cursor 0 fetches from the beginning. |
| 230 |
|
| 231 |
## Blob encryption |
| 232 |
|
| 233 |
Binary blobs (files, images, audio samples) are encrypted client-side before |
| 234 |
upload: |
| 235 |
|
| 236 |
1. `blob_upload_url(hash, plaintext_size)` requests a presigned S3 PUT URL from |
| 237 |
the server. If the blob already exists (content-addressed by hash), no upload |
| 238 |
is needed. The declared size is converted to the ciphertext length before it |
| 239 |
is sent, because the server signs it as `Content-Length` and the PUT carries |
| 240 |
sealed bytes. |
| 241 |
2. `blob_upload(hash, presigned_url, data)` encrypts the plaintext bytes with the |
| 242 |
master key, **binding the content hash as AEAD associated data**, and PUTs the |
| 243 |
ciphertext to S3. |
| 244 |
3. `blob_confirm(hash, size)` tells the server the upload completed. |
| 245 |
|
| 246 |
Steps 1 and 2 are the in-memory path, for bytes with no file behind them. Every |
| 247 |
consumer app instead calls `blob_upload_streaming(hash, path)` (below), which |
| 248 |
covers the same ground with bounded memory and no size ceiling, then step 3. |
| 249 |
4. `blob_download_url(hash)` requests a presigned GET URL. |
| 250 |
5. `blob_download(expected_hash, presigned_url)` fetches the ciphertext, decrypts |
| 251 |
with the hash as AAD, then **re-verifies** that the plaintext hashes to |
| 252 |
`expected_hash` (returning `IntegrityFailed` otherwise). The AEAD tag proves |
| 253 |
the bytes were not forged; the re-hash proves the server served the bytes for |
| 254 |
*this* address, closing the substitution/rollback gap. |
| 255 |
|
| 256 |
Blob encryption uses `encrypt_bytes_aad`/`decrypt_bytes_aad` (raw bytes, no |
| 257 |
base64) to avoid the ~33% base64 overhead on large files. v2 overhead is 44 bytes |
| 258 |
per blob: a 4-byte `sk2:` wire tag + 24-byte XChaCha20 nonce + 16-byte Poly1305 |
| 259 |
tag. |
| 260 |
|
| 261 |
### Large blobs: the multipart path |
| 262 |
|
| 263 |
Steps 1 and 2 buffer the whole ciphertext, and the server refuses a one-shot PUT |
| 264 |
above its own ceiling. For a blob already on disk, `blob_upload_streaming(hash, |
| 265 |
path)` replaces both with a multipart session and holds one part at a time: |
| 266 |
|
| 267 |
1. It computes the exact ciphertext length from the file size alone |
| 268 |
(`blob_encrypted_len`) and declares it to `blobs/multipart/start`, which |
| 269 |
returns the part geometry both sides then derive identically. |
| 270 |
2. It seals 1 MiB v3 chunks straight out of the file, staging them until a full |
| 271 |
part is ready, and PUTs each part to a presigned URL signed for that exact |
| 272 |
`Content-Length`. Each presigned part URL is requested on its own, one per |
| 273 |
part, since a part's checksum only exists once it has been sealed and only one |
| 274 |
part is ever held in memory. |
| 275 |
3. It recomputes the SHA-256 as it reads, so a file that changed since the |
| 276 |
caller hashed it fails rather than landing under a stale content address. |
| 277 |
4. Any failure past `start` aborts the session, since uploaded parts bill until |
| 278 |
released. `blob_confirm` is still the caller's next step, unchanged. |
| 279 |
|
| 280 |
It works at any size (a small blob is a single part), so prefer it whenever the |
| 281 |
blob is file-backed. Not yet resumable across a process restart: the session id |
| 282 |
is not persisted. |
| 283 |
|
| 284 |
## Keychain integration |
| 285 |
|
| 286 |
The `keychain` feature (default on) uses the `keyring` crate (v4) to store |
| 287 |
the master key in the OS credential store: |
| 288 |
|
| 289 |
- **macOS**: Keychain Services |
| 290 |
- **Linux**: secret-service (D-Bus Secret Service API) |
| 291 |
- **Windows**: Windows Credential Manager |
| 292 |
|
| 293 |
Keychain entries are keyed by `synckit:<app_id>` (service) and `<user_id>` |
| 294 |
(account). When the feature is disabled, store/load/delete are no-ops that |
| 295 |
return `Ok`. |
| 296 |
|
| 297 |
## Retry and resilience |
| 298 |
|
| 299 |
Push and pull use `retry_request()` with exponential backoff: |
| 300 |
|
| 301 |
- **Max retries**: 3 |
| 302 |
- **Delays**: 1s, 2s, 4s (BASE_DELAY * 2^attempt) |
| 303 |
- **Transient errors** (retried): network failures (timeout, DNS, connection |
| 304 |
refused), server errors (5xx), rate limiting (429) |
| 305 |
- **Permanent errors** (not retried): client errors (4xx except 429), |
| 306 |
serialization errors, encryption errors, missing session/key |
| 307 |
|
| 308 |
Token expiry is detected client-side by decoding the JWT `exp` claim with a |
| 309 |
30-second buffer. If the token is about to expire, the client returns |
| 310 |
`SyncKitError::TokenExpired` so the caller can re-authenticate before sending |
| 311 |
a request that would fail with 401. |
| 312 |
|
| 313 |
## Key design decisions |
| 314 |
|
| 315 |
### Why XChaCha20-Poly1305 |
| 316 |
|
| 317 |
- 192-bit nonces are large enough to generate randomly without realistic |
| 318 |
collision risk (unlike AES-GCM's 96-bit nonces). |
| 319 |
- Removes the need for a nonce counter or nonce-misuse resistance scheme. |
| 320 |
- Performance is comparable to AES-GCM on modern hardware. |
| 321 |
|
| 322 |
### Why Argon2id |
| 323 |
|
| 324 |
- OWASP-recommended for password hashing. Resists both side-channel (Argon2i) |
| 325 |
and GPU brute-force (Argon2d) attacks. |
| 326 |
- Parameters (64 MB memory, 3 iterations) meet the OWASP interactive minimum. |
| 327 |
|
| 328 |
### Why random salt per wrap |
| 329 |
|
| 330 |
Each `wrap_master_key` call generates a fresh 32-byte random salt. This means |
| 331 |
re-wrapping with the same password produces a completely different envelope, |
| 332 |
preventing precomputation attacks and ensuring password changes are |
| 333 |
cryptographically distinct. |
| 334 |
|
| 335 |
### Why no token refresh |
| 336 |
|
| 337 |
The server currently issues short-lived JWTs without refresh tokens. The client |
| 338 |
detects expiry and returns `TokenExpired`, leaving re-authentication to the |
| 339 |
caller. This keeps the SDK stateless with respect to refresh logic and avoids |
| 340 |
storing long-lived credentials in memory. |
| 341 |
|
| 342 |
## Security properties |
| 343 |
|
| 344 |
- **Server-zero-knowledge**: The server stores only ciphertext (envelope, sync |
| 345 |
entries, blobs). It never receives the plaintext master key or user data. |
| 346 |
- **Key zeroization**: The `ZeroizeOnDrop` wrapper uses volatile writes to |
| 347 |
clear the master key from memory when the guard is dropped. |
| 348 |
- **No key material in logs**: Tracing statements log events (key stored, key |
| 349 |
loaded) but never log key bytes or ciphertext. |
| 350 |
- **Minimum ciphertext size**: Decryption rejects inputs shorter than 40 bytes |
| 351 |
(24-byte nonce + 16-byte tag), preventing trivial malformed-input attacks. |
| 352 |
- **Ciphertext position binding (AAD)**: v2 entry ciphertext is sealed with |
| 353 |
`(table, row_id)` as AEAD associated data, and blob ciphertext with the content |
| 354 |
hash; downloaded blobs are additionally re-hashed and rejected if they do not |
| 355 |
match their requested address. So the untrusted server cannot relocate or |
| 356 |
substitute a valid ciphertext without the open (or the hash check) failing |
| 357 |
closed. See "Wire version tag and AAD binding". |
| 358 |
- **Envelope versioning**: The key envelope includes a version field (`v: 1`) |
| 359 |
and travels with its Argon2 cost parameters, so the work factor can be raised |
| 360 |
later without breaking existing envelopes. |
| 361 |
|
| 362 |
--- |
| 363 |
|
| 364 |
# The `SyncStore` higher-level helper |
| 365 |
|
| 366 |
> Status: shipped in 0.6 behind the default-on `store` feature (`src/store/`). |
| 367 |
> Everything above describes the base SDK (transport + crypto + HLC + conflict |
| 368 |
> primitives); this section describes the layer *above* those primitives that |
| 369 |
> absorbs the SQLite plumbing the three first-party apps otherwise hand-write. |
| 370 |
> The design was reverse-engineered from the GoingsOn, audiofiles, and Balanced |
| 371 |
> Breakfast sync services; a consumer that only needs the transport/crypto SDK |
| 372 |
> (e.g. mnw-cli) sets `default-features = false` and skips it. |
| 373 |
|
| 374 |
## Motivation |
| 375 |
|
| 376 |
Without this layer the SDK stops at "encrypt/decrypt a `Vec<ChangeEntry>` and |
| 377 |
move it over the wire," and each consuming app writes ~700–1,000 lines of |
| 378 |
*identical* engine on top of it: a `sync_changelog` + `sync_state` schema, per-table triggers, a |
| 379 |
drain-loop push, a cursor-loop pull, an FK-ordered apply with JSON→SQL binding, |
| 380 |
HLC stamping, an initial snapshot, retention/cleanup, and a scheduler |
| 381 |
(tick + SSE + backoff + status events). Measured across the three apps this is |
| 382 |
the overwhelming majority of their sync code, and it is mechanical. |
| 383 |
|
| 384 |
The single largest boilerplate source is **column-list triplication**: every |
| 385 |
syncable table names its synced columns three or four times, in the trigger DDL |
| 386 |
(`json_object(...)`), in the initial-snapshot projection, and in the apply-side |
| 387 |
whitelist, kept in agreement only by hand and a round-trip drift test. When they |
| 388 |
drift, a column silently arrives NULL (this is a real shipped bug class). One |
| 389 |
declared column list should generate all of them. |
| 390 |
|
| 391 |
The goal of `SyncStore` is: an app **declares its tables and policy once**, and |
| 392 |
the engine owns everything mechanical. This is the "simple" pillar of the product |
| 393 |
(see the private roadmap), the differentiators (private, secure, economical) are |
| 394 |
already in the 0.6 primitives; this layer is what makes them easy to adopt. |
| 395 |
|
| 396 |
## What is generic (absorbed) vs. what is policy (declared) |
| 397 |
|
| 398 |
The three apps agree, byte-for-byte in spirit, on all of this, it moves into the |
| 399 |
engine: |
| 400 |
|
| 401 |
- `sync_state` KV + the transactional `applying_remote` echo-suppression flag. |
| 402 |
- `sync_changelog` schema, the `pushed` flag, cursor bookkeeping. |
| 403 |
- Push: HLC-stamp unpushed rows, batch ≤500, map to `ChangeEntry`, `client.push`, |
| 404 |
mark pushed, skip-and-retain corrupt/unknown-op rows, drain until empty. |
| 405 |
- Pull: paginated `client.pull_rich` loop, apply-then-persist-cursor (the SDK's |
| 406 |
documented crash-safe contract), auto-decrypt. |
| 407 |
- Apply: partition into upserts/deletes, iterate the declared table order |
| 408 |
(parents-first upserts, children-first deletes), JSON→SQL type coercion, |
| 409 |
per-row constraint-violation *skip-and-log* (a poison row never wedges the |
| 410 |
cursor), HLC gating against committed clocks, `resolve_lww` for true conflicts. |
| 411 |
- Initial snapshot, `cleanup_changelog` (prune pushed > N days), changelog |
| 412 |
retention cap, pending-change count. |
| 413 |
- The scheduler skeleton: 60 s tick raced against the SSE `changed` stream, the |
| 414 |
auth/key/enabled gate chain, a shared sync mutex, exponential backoff, and the |
| 415 |
402 → subscription-required special case. |
| 416 |
- Device registration and the `perform_sync` orchestration order. |
| 417 |
|
| 418 |
Everything an app genuinely varies is small and enumerable. It becomes a |
| 419 |
declarative `SyncSchema` plus a couple of trait hooks: |
| 420 |
|
| 421 |
|
| 422 |
|
| 423 |
| Which tables, their columns, FK order | all | `SyncSchema { tables: [SyncTable…] }`, declaration order = FK order | |
| 424 |
| Primary key shape (single / composite) | BB tag tables, AF `tags` | `PrimaryKey::{Single, Composite}` | |
| 425 |
| Partial-column, UPDATE-only sync | BB `feed_items` (is_read/is_starred) | `SyncMode::PartialUpdate { set }` | |
| 426 |
| Ignore remote deletes | BB `feed_items` | `DeleteMode::Ignore` | |
| 427 |
| Soft-delete (tombstone) instead of hard | AF `samples.deleted_at` (CASCADE safety) | `DeleteMode::Tombstone { column }` | |
| 428 |
| Preserve local-only columns on upsert | GO `email_accounts` credentials | `preserve_local: &[col]` | |
| 429 |
| Column defaults on first insert | GO `password=''` for NOT NULL | `insert_defaults: &[(col, val)]` | |
| 430 |
| Row references an *unsynced* table (relax FK) | GO `tasks.source_email_id` | `references_unsynced: true` | |
| 431 |
| Row-id privacy hashing on the wire | AF content tables (`hash_row_id(salt,…)`) | `RowIdScheme::Hashed` | |
| 432 |
| Exclude some rows/keys from sync | AF `user_config` `sync_*`, `loose_files` | `exclude_where: Option<&str>` (a SQL predicate, see AF example; must be SQL because it also compiles into the generated trigger's WHEN clause, not just the apply guard) | |
| 433 |
| Conflict model | GO/AF (HLC), BB (server-order) | `ConflictStrategy::{HybridLogicalClock, ServerOrder}` | |
| 434 |
| Blob-bearing tables | AF only | `BlobPolicy` trait (below) | |
| 435 |
| Retention cap, cleanup window, batch, interval | all (different numbers) | `SyncConfig` | |
| 436 |
| Device-name source, status sink, error mapping | all (framework-specific) | `SyncObserver` trait | |
| 437 |
|
| 438 |
User scoping (a multi-tenant `WHERE user_id = ?` filter) is deliberately **not** |
| 439 |
an engine-level concern. The GoingsOn worked example (below) confirmed the single- |
| 440 |
user desktop apps never scope the *changelog* by user, the only place a user |
| 441 |
filter appears is inside GO's blob queries, which is policy-local to its |
| 442 |
`BlobPolicy`. A changelog-level user filter belongs to the future multi-tenant |
| 443 |
work (a third-party-phase prerequisite), not this layer. |
| 444 |
|
| 445 |
## The declarative schema |
| 446 |
|
| 447 |
```rust |
| 448 |
// Abridged from the shipped `store::schema` types. The real fields are private |
| 449 |
// (`pub(crate)`) and constructed through the `SyncSchema`/`SyncTable` builders |
| 450 |
// (`.table(...)`, `.pk(&[...])`, `.conflict(...)`); shown here public for shape. |
| 451 |
pub struct SyncSchema { |
| 452 |
/// Declaration order IS the FK order: parents first. The engine upserts in |
| 453 |
/// this order and deletes in the reverse. No hand-maintained reversed list. |
| 454 |
pub tables: Vec<SyncTable>, |
| 455 |
/// Schema-wide conflict model (HLC or server-order). |
| 456 |
pub conflict: ConflictStrategy, |
| 457 |
} |
| 458 |
|
| 459 |
pub struct SyncTable { |
| 460 |
pub name: &'static str, |
| 461 |
/// The one column whitelist. Drives generated triggers, the snapshot |
| 462 |
/// projection, and apply-side binding — killing the triplication. |
| 463 |
pub columns: &'static [&'static str], |
| 464 |
/// PK column(s): one element = single key, more = composite (concatenated |
| 465 |
/// with ':' for the wire row-id). A slice, not a Single/Composite enum — |
| 466 |
/// simpler and matches the `.pk(&[...])` builder. Defaults to `["id"]`. |
| 467 |
pub pk: &'static [&'static str], |
| 468 |
pub mode: SyncMode, |
| 469 |
pub deletes: DeleteMode, |
| 470 |
pub row_id: RowIdScheme, |
| 471 |
/// Columns to leave untouched on upsert (secrets that stay per-device). |
| 472 |
pub preserve_local: &'static [&'static str], |
| 473 |
/// Values to inject only on first INSERT (satisfy NOT NULL on preserved cols). |
| 474 |
pub insert_defaults: &'static [(&'static str, &'static str)], |
| 475 |
/// This table carries FKs into a table that is not synced; relax FK |
| 476 |
/// enforcement while applying it. |
| 477 |
pub references_unsynced: bool, |
| 478 |
/// Optional SQL predicate that excludes rows from sync (AF config-key |
| 479 |
/// denylist). A SQL string, not a closure, because it also compiles into |
| 480 |
/// the generated trigger's WHEN clause — not just the apply-side guard. |
| 481 |
pub exclude_where: Option<&'static str>, |
| 482 |
} |
| 483 |
|
| 484 |
pub enum SyncMode { |
| 485 |
/// Upsert every whitelisted column (INSERT … ON CONFLICT DO UPDATE). |
| 486 |
Full, |
| 487 |
/// UPDATE only `set` columns where PK matches; never insert, never replace. |
| 488 |
/// BB feed_items: sync just is_read/is_starred, never the article body. |
| 489 |
PartialUpdate { set: &'static [&'static str] }, |
| 490 |
} |
| 491 |
|
| 492 |
pub enum DeleteMode { |
| 493 |
Hard, |
| 494 |
/// Remote deletes are dropped on the floor (BB feed_items — content |
| 495 |
/// re-fetches from source). |
| 496 |
Ignore, |
| 497 |
/// Apply as `UPDATE … SET <column> = now()` instead of DELETE, so local |
| 498 |
/// ON DELETE CASCADE can't wipe organization (AF samples). |
| 499 |
Tombstone { column: &'static str }, |
| 500 |
} |
| 501 |
|
| 502 |
pub enum RowIdScheme { |
| 503 |
/// Wire row_id is the cleartext primary key (GO, BB). |
| 504 |
PrimaryKey, |
| 505 |
/// Wire row_id is a salted hash of the PK; the cleartext PK is carried |
| 506 |
/// inside the (encrypted) payload for DELETEs (AF content tables). Requires |
| 507 |
/// three engine duties: own a per-vault `row_id_salt` in sync_state (generate |
| 508 |
/// once, never sync), register the `hash_row_id` SQL function on the |
| 509 |
/// connection, and make the generated DELETE trigger emit the PK column(s) |
| 510 |
/// into `data` (so pull reconstructs the WHERE from the payload, never the |
| 511 |
/// opaque row_id). Assigned per-table: content-bearing keys are Hashed, |
| 512 |
/// opaque-integer / closed-config keys stay PrimaryKey. |
| 513 |
Hashed, |
| 514 |
} |
| 515 |
``` |
| 516 |
|
| 517 |
Note the FK-order simplification: today every app keeps `UPSERT_ORDER` *and* a |
| 518 |
hand-written `DELETE_ORDER` that a test asserts is its exact reverse. Declaration |
| 519 |
order supplies both. |
| 520 |
|
| 521 |
## Generated migrations (the triplication killer) |
| 522 |
|
| 523 |
```rust |
| 524 |
impl SyncSchema { |
| 525 |
/// Emit the full sync DDL: sync_state, sync_changelog, the HLC ledger tables, |
| 526 |
/// and every table's INSERT/UPDATE/DELETE trigger — the applying_remote guard, |
| 527 |
/// the json_object projection over `columns`, the extra OLD≠NEW guard for |
| 528 |
/// PartialUpdate tables, and the row_id expression (cleartext or hashed) — all |
| 529 |
/// derived from the same declaration the apply side reads. |
| 530 |
pub fn migration_sql(&self) -> String; |
| 531 |
} |
| 532 |
``` |
| 533 |
|
| 534 |
An app runs `migration_sql()` once as a normal migration. Because triggers, |
| 535 |
snapshot, and apply all read one `columns` list, the drift class is eliminated by |
| 536 |
construction and the round-trip test each app carries today is deleted. |
| 537 |
|
| 538 |
## The engine and its database seam |
| 539 |
|
| 540 |
The one real portability constraint: GO and BB drive SQLite through an async |
| 541 |
`sqlx::SqlitePool`; audiofiles uses `rusqlite` behind `spawn_blocking` (its |
| 542 |
`Connection` is `!Send`). The engine must not marry either driver. |
| 543 |
|
| 544 |
Resolution: **`SyncStore` owns its own `rusqlite` connection(s) to the app's |
| 545 |
SQLite file** (given a path or a connection factory). Under WAL, which all three |
| 546 |
apps already use, the engine's short read/apply transactions coexist with the |
| 547 |
app's own pool on the same file. The app's domain code is untouched; the sync |
| 548 |
engine simply becomes another connection that reads `sync_changelog` and writes |
| 549 |
the domain tables under the `applying_remote` guard. This also matches audiofiles |
| 550 |
exactly and removes GO/BB's async-sync code entirely. |
| 551 |
|
| 552 |
Per-connection state the engine controls on its own connection: `PRAGMA |
| 553 |
foreign_keys` (toggled off only while applying a `references_unsynced` table, then |
| 554 |
restored, or the connection dropped rather than returned in an unknown state), |
| 555 |
and the transaction that raises/lowers `applying_remote` *inside* the same commit |
| 556 |
as the writes (so a crash rolls the flag back, never stranding it at `'1'` and |
| 557 |
silently swallowing later local edits). |
| 558 |
|
| 559 |
```rust |
| 560 |
pub struct SyncStore { /* schema, client, config, db factory, blob policy, observer */ } |
| 561 |
|
| 562 |
impl SyncStore { |
| 563 |
pub fn builder(db: DbSource, client: SyncKitClient, schema: SyncSchema) -> SyncStoreBuilder; |
| 564 |
|
| 565 |
/// One full cycle: ensure device → push → pull → (blobs) → stamp last_sync. |
| 566 |
pub async fn sync_now(&self) -> Result<SyncOutcome>; |
| 567 |
|
| 568 |
/// Own the whole background loop (tick + SSE + gates + backoff), reporting |
| 569 |
/// through the SyncObserver. Returns a handle to stop it. |
| 570 |
pub fn spawn_scheduler(&self, observer: Arc<dyn SyncObserver>) -> SchedulerHandle; |
| 571 |
|
| 572 |
pub async fn pending_changes(&self) -> Result<u64>; |
| 573 |
pub async fn disconnect(&self) -> Result<()>; |
| 574 |
} |
| 575 |
|
| 576 |
pub struct SyncOutcome { pub pushed: u64, pub pulled: u64, pub changed_tables: HashSet<String> } |
| 577 |
``` |
| 578 |
|
| 579 |
## Conflict strategy |
| 580 |
|
| 581 |
GO and AF run the full HLC pipeline (`detect_conflicts` → committed-HLC gating → |
| 582 |
`resolve_lww`); BB currently sends a nil HLC and leans on server ordering. Rather |
| 583 |
than force a migration, expose it: |
| 584 |
|
| 585 |
```rust |
| 586 |
pub enum ConflictStrategy { |
| 587 |
/// Stamp a real HLC per edit, gate clean changes against committed clocks, |
| 588 |
/// resolve true conflicts with resolve_lww. Deterministic multi-device |
| 589 |
/// convergence. The default and recommended setting. |
| 590 |
HybridLogicalClock, |
| 591 |
/// Trust server seq order; last delivered wins. Simplest; adequate when |
| 592 |
/// concurrent edits to the same row are rare/uninteresting. |
| 593 |
ServerOrder, |
| 594 |
} |
| 595 |
``` |
| 596 |
|
| 597 |
Default `HybridLogicalClock`; BB can adopt it (it is the roadmap intent anyway) or |
| 598 |
stay on `ServerOrder` during migration. |
| 599 |
|
| 600 |
## Blob policy |
| 601 |
|
| 602 |
Only audiofiles syncs blobs, but the shape is general: some rows own a |
| 603 |
content-addressed binary, gated by an enable flag, with a local/cloud presence |
| 604 |
bit and an in-place escape hatch. Expressed as a trait the engine drives *after* |
| 605 |
the metadata pull, non-fatally: |
| 606 |
|
| 607 |
```rust |
| 608 |
pub trait BlobPolicy: Send + Sync { |
| 609 |
/// Rows whose bytes should be uploaded (AF: samples in a sync_files VFS, |
| 610 |
/// not cloud_only, with a local file present). The engine skips any whose |
| 611 |
/// local_path() is absent, so this may return the full candidate set. |
| 612 |
fn pending_uploads(&self, conn: &Connection) -> Result<Vec<BlobRef>>; |
| 613 |
/// Rows whose bytes are missing locally and wanted (AF: sync_files VFS, |
| 614 |
/// source_path IS NULL). The engine skips any whose local_path() exists. |
| 615 |
fn pending_downloads(&self, conn: &Connection) -> Result<Vec<BlobRef>>; |
| 616 |
fn local_path(&self, blob: &BlobRef) -> PathBuf; |
| 617 |
|
| 618 |
// Presence bookkeeping — default no-ops. An app that tracks local-vs-cloud |
| 619 |
// presence purely by disk existence (GoingsOn) implements none of these; |
| 620 |
// an app with an explicit presence flag (AF cloud_only) overrides them. |
| 621 |
/// Reconcile presence flags before the passes (AF: mark_cloud_only_samples). |
| 622 |
fn reconcile(&self, _conn: &Connection) -> Result<()> { Ok(()) } |
| 623 |
/// Flip bookkeeping after upload, echo-suppressed (AF: cloud_only). |
| 624 |
fn on_uploaded(&self, _conn: &Connection, _blob: &BlobRef) -> Result<()> { Ok(()) } |
| 625 |
/// Flip bookkeeping after download, echo-suppressed (AF: cloud_only). |
| 626 |
fn on_downloaded(&self, _conn: &Connection, _blob: &BlobRef) -> Result<()> { Ok(()) } |
| 627 |
} |
| 628 |
|
| 629 |
pub struct BlobRef { pub hash: String, pub ext: String, pub size: u64 } |
| 630 |
``` |
| 631 |
|
| 632 |
The engine owns the invariant parts: the skip-if-`local_path`-absent → |
| 633 |
`upload_file` (which streams and dedups server-side) → `blob_confirm` dance, the |
| 634 |
`blob_download_url` → skip-if-`local_path`-present → `blob_download` → **SHA-256 |
| 635 |
re-verify** → atomic temp+rename write, and the single-item `download_one(hash)` |
| 636 |
path for a GUI "download now" action. The policy trait only answers "which blobs," |
| 637 |
"where on disk," and (optionally) "how to reflect presence." GoingsOn's attachment |
| 638 |
policy needs only the first three methods; audiofiles overrides all six. |
| 639 |
|
| 640 |
## Status reporting |
| 641 |
|
| 642 |
Status must be framework-agnostic: GO/BB emit Tauri events, AF mutates an |
| 643 |
`Arc<Mutex<SyncStatus>>`. A trait bridges both: |
| 644 |
|
| 645 |
```rust |
| 646 |
pub trait SyncObserver: Send + Sync { |
| 647 |
fn on_status(&self, status: SyncStatus); // Syncing/Idle/Error/SubscriptionRequired/LoggedOut |
| 648 |
fn on_changes_applied(&self, tables: &HashSet<String>); // selective cache invalidation |
| 649 |
fn on_subscription_required(&self); |
| 650 |
} |
| 651 |
``` |
| 652 |
|
| 653 |
## What each app sheds |
| 654 |
|
| 655 |
|
| 656 |
|
| 657 |
| **GoingsOn** | ~1,400 non-test LOC across push/pull/apply/hlc/state/blob_sync/scheduler + trigger DDL + drift test | A 20-table `SyncSchema`, `email_accounts` `preserve_local`+`insert_defaults`, `references_unsynced` on `tasks`+`attachments`, a thin attachment `BlobPolicy`, a Tauri `SyncObserver` (see worked example below) | |
| 658 |
| **audiofiles** | ~1,400 non-test LOC + the whole blob subsystem + tombstone/hash-id/config-exclusion logic | A 16-table `SyncSchema` (per-table `Hashed` row-ids, `Tombstone` on samples, `exclude_where` on user_config, composite/all-PK tables), the full 5-method `BlobPolicy` impl (kept, genuine policy), an `Arc<Mutex>` `SyncObserver` (see worked example below) | |
| 659 |
| **Balanced Breakfast** | ~700 non-test LOC + triplicated columns | A 7-table `SyncSchema` with `feed_items = PartialUpdate{set:[is_read,is_starred]} + DeleteMode::Ignore`, `MAX_CHANGELOG_ENTRIES` in `SyncConfig`, a Tauri `SyncObserver`. Adopts `HybridLogicalClock` | |
| 660 |
|
| 661 |
## Design decisions |
| 662 |
|
| 663 |
Decided (2026-07-12): |
| 664 |
|
| 665 |
1. **DB ownership: the engine owns a private `rusqlite` connection to the app's |
| 666 |
WAL file.** Not a `SyncDb` trait over the app's `sqlx` pool. This is far |
| 667 |
simpler, matches audiofiles exactly, and lets GoingsOn and Balanced Breakfast |
| 668 |
delete their async sync code wholesale. The engine's short read/apply |
| 669 |
transactions coexist with the app's own pool under WAL. Consequence: the app |
| 670 |
passes a path (or a connection factory for custom pragmas/shared-cache), and |
| 671 |
the engine, not the app, holds the FK-pragma and `applying_remote` state on its |
| 672 |
own connection. |
| 673 |
2. **Triggers are generated from the manifest**, not hand-written SQL. One |
| 674 |
`columns` list per table produces the trigger DDL, the snapshot projection, and |
| 675 |
the apply binding, so the column-drift bug class is closed by construction and |
| 676 |
the per-app round-trip drift test is deleted. Accepted cost: the engine emits |
| 677 |
SQL the app reviews once at migration time. |
| 678 |
|
| 679 |
Remaining to confirm during build (not blocking): |
| 680 |
|
| 681 |
3. **Null-tolerance merge policy** (omit NOT-NULL-null columns, keep nullable-null) |
| 682 |
reads the live schema, so it defaults into the engine; confirm no app needs a |
| 683 |
different per-table rule. |
| 684 |
4. **`extra`/forward-compat fields**, the apply path must thread |
| 685 |
`ChangeEntry.extra` through unchanged so a newer client's fields survive an |
| 686 |
older client's round-trip; confirm nothing drops them. |
| 687 |
|
| 688 |
## Worked example: GoingsOn |
| 689 |
|
| 690 |
GoingsOn is the richest consumer (20 tables, credential exclusion, cross-table |
| 691 |
FKs into unsynced data, blobs, HLC), so it is the design's stress test. Its entire |
| 692 |
sync engine, `push.rs`, `pull.rs`, `apply.rs`, `hlc.rs`, `state.rs`, `blob_sync.rs`, |
| 693 |
most of `mod.rs`, the hand-written trigger migrations, and the drift test, reduces |
| 694 |
to the manifest and one thin trait impl below. Column lists are the *actual* |
| 695 |
`SYNCED_COLUMNS` from `apply.rs`. |
| 696 |
|
| 697 |
```rust |
| 698 |
use synckit_client::store::{SyncSchema, SyncTable, ConflictStrategy}; |
| 699 |
|
| 700 |
// Common case: single `id` PK, cleartext wire row-id, Full upsert (ON CONFLICT |
| 701 |
// DO UPDATE — never INSERT OR REPLACE, so ON DELETE CASCADE can't wipe children), |
| 702 |
// Hard delete, HLC conflict resolution. 17 of GO's 20 tables are one line each. |
| 703 |
fn t(name: &'static str, cols: &'static [&'static str]) -> SyncTable { |
| 704 |
SyncTable::full(name, cols) |
| 705 |
} |
| 706 |
|
| 707 |
pub fn goingson_schema() -> SyncSchema { |
| 708 |
// Declaration order = FK order (parents first). Deletes run the exact reverse |
| 709 |
// automatically — no hand-maintained DELETE_ORDER, no test asserting it. |
| 710 |
SyncSchema::new(vec![ |
| 711 |
t("projects", &["id","name","description","project_type","status","created_at","user_id"]), |
| 712 |
t("contacts", &["id","user_id","display_name","nickname","company","title","notes","tags", |
| 713 |
"birthday","timezone","external_source","external_id","created_at","updated_at"]), |
| 714 |
|
| 715 |
// email_accounts: sync config, NEVER credentials. GO's bespoke ~40-line |
| 716 |
// apply_email_account_upsert() dissolves into these two declarative fields: |
| 717 |
// preserve_local → the 4 secret columns are untouched on conflict-update |
| 718 |
// insert_defaults → new rows get password='' to satisfy its NOT NULL |
| 719 |
SyncTable::full("email_accounts", |
| 720 |
&["id","user_id","account_name","email_address","imap_server","imap_port", |
| 721 |
"smtp_server","smtp_port","username","use_tls","created_at","archive_folder_name", |
| 722 |
"auth_type","jmap_session_url","jmap_account_id","sync_interval_minutes","email_signature"]) |
| 723 |
.preserve_local(&["password","oauth2_access_token","oauth2_refresh_token","oauth2_token_expires_at"]) |
| 724 |
.insert_defaults(&[("password", "")]), |
| 725 |
|
| 726 |
t("sync_accounts", &["id","user_id","provider","account_name","email","sync_calendars", |
| 727 |
"sync_contacts","calendar_ids","sync_interval_minutes","enabled","created_at"]), |
| 728 |
t("milestones", &["id","user_id","project_id","name","description","position","target_date", |
| 729 |
"status","created_at"]), |
| 730 |
|
| 731 |
// tasks & attachments both carry source_email_id — a FK into the emails |
| 732 |
// table, which is NOT synced (emails stay per-device). Relax FK enforcement |
| 733 |
// while applying them. (In GO today this is a blanket PRAGMA foreign_keys=OFF |
| 734 |
// over the whole apply; the engine does the same when any batched table is |
| 735 |
// references_unsynced.) |
| 736 |
SyncTable::full("tasks", |
| 737 |
&["id","project_id","description","status","priority","due","tags","urgency","recurrence", |
| 738 |
"recurrence_rule","created_at","user_id","recurrence_parent_id","source_email_id", |
| 739 |
"snoozed_until","waiting_for_response","waiting_since","expected_response_date", |
| 740 |
"scheduled_start","scheduled_duration","is_focus","focus_set_at","contact_id", |
| 741 |
"milestone_id","completed_at","estimated_minutes","actual_minutes"]) |
| 742 |
.references_unsynced(), |
| 743 |
t("time_sessions", &["id","task_id","user_id","started_at","ended_at","duration_minutes","created_at"]), |
| 744 |
SyncTable::full("attachments", |
| 745 |
&["id","user_id","task_id","project_id","filename","file_size","mime_type","blob_hash", |
| 746 |
"source_email_id","created_at"]) |
| 747 |
.references_unsynced(), |
| 748 |
|
| 749 |
t("events", &["id","project_id","title","description","start_time","end_time","location","user_id", |
| 750 |
"linked_task_id","recurrence","recurrence_parent_id","recurrence_rule","contact_id", |
| 751 |
"block_type","external_source","external_id","is_read_only","snoozed_until", |
| 752 |
"reminder_offsets_seconds"]), |
| 753 |
t("annotations", &["id","task_id","timestamp","note"]), |
| 754 |
t("subtasks", &["id","task_id","text","is_completed","position","created_at","linked_task_id"]), |
| 755 |
t("contact_emails", &["id","contact_id","address","label","is_primary"]), |
| 756 |
t("contact_phones", &["id","contact_id","number","label","is_primary"]), |
| 757 |
t("contact_social_handles", &["id","contact_id","platform","handle","url"]), |
| 758 |
t("contact_custom_fields", &["id","contact_id","label","value","url"]), |
| 759 |
t("daily_notes", &["id","user_id","note_date","went_well","could_improve","is_reviewed", |
| 760 |
"reviewed_at","created_at","updated_at"]), |
| 761 |
t("saved_views", &["id","user_id","name","view_type","filters","sort_by","sort_order", |
| 762 |
"is_pinned","position","created_at","updated_at"]), |
| 763 |
t("weekly_reviews", &["id","user_id","week_start_date","completed_at","notes","vacation_days"]), |
| 764 |
t("monthly_goals", &["id","user_id","month","text","status","position","created_at","updated_at"]), |
| 765 |
t("monthly_reflections", &["id","user_id","month","highlight_text","change_text","completed_at"]), |
| 766 |
]) |
| 767 |
.conflict_strategy(ConflictStrategy::HybridLogicalClock) // GO's current model; also the default |
| 768 |
} |
| 769 |
``` |
| 770 |
|
| 771 |
The blob policy is thin, GO tracks local-vs-cloud presence purely by disk |
| 772 |
existence (no `cloud_only` flag), so it implements only the three required methods |
| 773 |
and inherits the no-op presence hooks: |
| 774 |
|
| 775 |
```rust |
| 776 |
struct AttachmentBlobs { user_id: String, data_dir: PathBuf } // user_id = DESKTOP_USER_ID |
| 777 |
|
| 778 |
impl BlobPolicy for AttachmentBlobs { |
| 779 |
fn pending_uploads(&self, conn: &Connection) -> Result<Vec<BlobRef>> { |
| 780 |
// SELECT DISTINCT blob_hash, file_size FROM attachments WHERE user_id = ? |
| 781 |
// (engine skips any whose local_path is absent) |
| 782 |
} |
| 783 |
fn pending_downloads(&self, conn: &Connection) -> Result<Vec<BlobRef>> { |
| 784 |
// SELECT DISTINCT blob_hash FROM attachments WHERE user_id = ? |
| 785 |
// (engine skips any whose local_path already exists; then SHA-256-verifies) |
| 786 |
} |
| 787 |
fn local_path(&self, blob: &BlobRef) -> PathBuf { blob_path(&self.data_dir, &blob.hash) } |
| 788 |
// reconcile / on_uploaded / on_downloaded: inherited no-ops. |
| 789 |
} |
| 790 |
``` |
| 791 |
|
| 792 |
### What GO proved about the design |
| 793 |
|
| 794 |
- **Confirmed: credential exclusion collapses.** The single most bespoke piece of |
| 795 |
GO's apply layer (`apply_email_account_upsert`, its own INSERT-with-`password=''` |
| 796 |
and 16-column ON CONFLICT) is fully expressed by `preserve_local` + |
| 797 |
`insert_defaults`. No per-table override function survives. This was the strongest |
| 798 |
signal the enum set is at the right altitude. |
| 799 |
- **Confirmed: 17 of 20 tables are one line.** Only `email_accounts`, `tasks`, and |
| 800 |
`attachments` need a builder call; the rest are `t(name, cols)`. |
| 801 |
- **Refinement 1: `user_scope` is not an engine concept.** GO never filters the |
| 802 |
*changelog* by user; the only `WHERE user_id = ?` is inside its blob queries. |
| 803 |
So user scoping is policy-local (it lives in `AttachmentBlobs`), and the engine- |
| 804 |
level `user_scope` field was removed. A changelog-level tenant filter is deferred |
| 805 |
to the future multi-tenant work. |
| 806 |
- **Refinement 2: blob presence hooks must default to no-ops.** GO has no |
| 807 |
`cloud_only` equivalent, so `reconcile`/`on_uploaded`/`on_downloaded` gained |
| 808 |
default empty bodies; only AF overrides them. |
| 809 |
- **Corrected: `references_unsynced` covers `attachments` too**, not just `tasks` |
| 810 |
(both carry `source_email_id`). The engine's rule "disable FK for the apply txn if |
| 811 |
any batched table is `references_unsynced`" matches GO's existing blanket toggle. |
| 812 |
|
| 813 |
## Worked example: audiofiles |
| 814 |
|
| 815 |
audiofiles is the design's hardest test: it exercises everything GoingsOn does not |
| 816 |
, salted-hash wire row-ids, tombstone deletes, a SQL exclusion predicate, composite |
| 817 |
and all-PK tables, and the full six-touch `BlobPolicy`. Column lists, PKs, and |
| 818 |
orders are the actual `table_columns`/`pk_columns` from `service/mod.rs`. |
| 819 |
|
| 820 |
```rust |
| 821 |
use synckit_client::store::{SyncSchema, SyncTable}; |
| 822 |
|
| 823 |
// Cleartext single-`id` PK, Full upsert, Hard delete (opaque-integer tables that |
| 824 |
// carry no user content in their key — never row-id-hashed per M018). |
| 825 |
fn t(name: &'static str, cols: &'static [&'static str]) -> SyncTable { |
| 826 |
SyncTable::full(name, cols) |
| 827 |
} |
| 828 |
|
| 829 |
pub fn audiofiles_schema() -> SyncSchema { |
| 830 |
SyncSchema::new(vec![ |
| 831 |
t("vfs", &["id","name","created_at","modified_at","sync_files"]), |
| 832 |
|
| 833 |
// samples: stacks THREE modifiers — its PK is a content hash, so the wire |
| 834 |
// row-id is salted-hashed; and a remote delete must be a tombstone, never a |
| 835 |
// hard DELETE (vfs_nodes/tags/collection_members carry ON DELETE CASCADE |
| 836 |
// that applying_remote can't suppress). |
| 837 |
SyncTable::full("samples", |
| 838 |
&["hash","original_name","file_extension","file_size","import_date", |
| 839 |
"last_modified","cloud_only","duration"]) |
| 840 |
.pk(&["hash"]) |
| 841 |
.hashed() |
| 842 |
.tombstone("deleted_at"), |
| 843 |
|
| 844 |
SyncTable::full("sample_features", &["hash","feat_version","vector","computed_at"]) |
| 845 |
.pk(&["hash"]).hashed(), |
| 846 |
t("collections", &["id","name","description","created_at","filter_json"]), |
| 847 |
t("vfs_nodes", &["id","vfs_id","parent_id","name","node_type","sample_hash","created_at"]), |
| 848 |
SyncTable::full("audio_analysis", |
| 849 |
&["hash","bpm","musical_key","duration","sample_rate","channels","peak_db","rms_db", |
| 850 |
"is_loop","spectral_centroid","onset_strength","analyzed_at","lufs","spectral_flatness", |
| 851 |
"spectral_rolloff","zero_crossing_rate","classification","spectral_bandwidth", |
| 852 |
"centroid_variance","crest_factor","attack_time","classification_confidence"]) |
| 853 |
.pk(&["hash"]).hashed(), |
| 854 |
|
| 855 |
// tags: composite PK AND all columns are PK → engine emits INSERT OR IGNORE; |
| 856 |
// both key columns are user content → hashed. |
| 857 |
SyncTable::full("tags", &["sample_hash","tag"]).pk(&["sample_hash","tag"]).hashed(), |
| 858 |
SyncTable::full("tag_provenance", &["sample_hash","tag","source","rule_id"]) |
| 859 |
.pk(&["sample_hash","tag"]).hashed(), |
| 860 |
t("tag_rules", &["id","name","enabled","priority","match_mode","conditions","actions","created_at"]), |
| 861 |
SyncTable::full("tag_policy", &["tag","review_threshold","auto_threshold"]) |
| 862 |
.pk(&["tag"]).hashed(), |
| 863 |
t("classifier_layers", &["id","name","kind","weight","enabled","source","imported_at"]), |
| 864 |
t("classifier_exemplars", &["id","layer_id","feat_version","vector","tags"]), |
| 865 |
SyncTable::full("classifier_layer_rules", &["layer_id","rule_id"]).pk(&["layer_id","rule_id"]), |
| 866 |
SyncTable::full("collection_members", &["collection_id","sample_hash","added_at"]) |
| 867 |
.pk(&["collection_id","sample_hash"]).hashed(), |
| 868 |
|
| 869 |
// user_config: a SQL exclusion predicate applied symmetrically — it compiles |
| 870 |
// into the generated trigger WHEN (export) and the apply guard (import), so |
| 871 |
// sync-internal keys and the loose_files toggle never cross in either direction. |
| 872 |
SyncTable::full("user_config", &["key","value"]) |
| 873 |
.pk(&["key"]) |
| 874 |
// {row} → NEW/OLD per generated trigger (INSERT/UPDATE use NEW, DELETE uses OLD). |
| 875 |
.exclude_where("{row}.key NOT LIKE 'sync\\_%' ESCAPE '\\' AND {row}.key <> 'loose_files'"), |
| 876 |
|
| 877 |
t("edit_history", &["id","source_hash","result_hash","operation","params_json","created_at"]), |
| 878 |
]) |
| 879 |
// ConflictStrategy::HybridLogicalClock is the default; AF uses it. |
| 880 |
} |
| 881 |
``` |
| 882 |
|
| 883 |
The blob policy is the full implementation, AF tracks presence with an explicit |
| 884 |
`cloud_only` flag, so it overrides the presence hooks GO left as no-ops: |
| 885 |
|
| 886 |
```rust |
| 887 |
struct SampleBlobs { content_dir: PathBuf } |
| 888 |
|
| 889 |
impl BlobPolicy for SampleBlobs { |
| 890 |
fn pending_uploads(&self, conn: &Connection) -> Result<Vec<BlobRef>> { |
| 891 |
// samples in a sync_files VFS, blob present locally (cloud_only = 0): |
| 892 |
// SELECT DISTINCT s.hash, s.file_extension, s.file_size FROM samples s |
| 893 |
// JOIN vfs_nodes vn ON vn.sample_hash=s.hash JOIN vfs v ON v.id=vn.vfs_id |
| 894 |
// WHERE v.sync_files = 1 AND s.cloud_only = 0 |
| 895 |
} |
| 896 |
fn pending_downloads(&self, conn: &Connection) -> Result<Vec<BlobRef>> { |
| 897 |
// samples in a sync_files VFS not referenced in place: |
| 898 |
// … WHERE v.sync_files = 1 AND s.source_path IS NULL (engine skips present files) |
| 899 |
} |
| 900 |
fn local_path(&self, b: &BlobRef) -> PathBuf { self.content_dir.join(format!("{}.{}", b.hash, b.ext)) } |
| 901 |
|
| 902 |
// Overridden — the cloud_only presence flag (all echo-suppressed): |
| 903 |
fn reconcile(&self, conn: &Connection) -> Result<()> { |
| 904 |
// mark_cloud_only_samples: for cloud_only=0 AND source_path IS NULL samples |
| 905 |
// whose file is missing on disk, set cloud_only = 1. |
| 906 |
} |
| 907 |
fn on_downloaded(&self, conn: &Connection, b: &BlobRef) -> Result<()> { |
| 908 |
// clear cloud_only for b.hash after a verified fetch lands the file. |
| 909 |
} |
| 910 |
// on_uploaded: inherited no-op — AF's cloud_only is managed by reconcile+download, |
| 911 |
// not by upload (an uploaded sample was already local, i.e. cloud_only=0). |
| 912 |
} |
| 913 |
``` |
| 914 |
|
| 915 |
### What AF proved about the design |
| 916 |
|
| 917 |
- **Confirmed: the hard cases express.** `DeleteMode::Tombstone{column}` captures |
| 918 |
the samples CASCADE-safety rule; `PrimaryKey::Composite` + the engine's |
| 919 |
all-columns-are-PK → `INSERT OR IGNORE` detection handles `tags`/ |
| 920 |
`classifier_layer_rules`; and `samples` stacking `hashed()` + `tombstone()` + |
| 921 |
a non-`id` PK in one declaration validates that the modifiers compose. |
| 922 |
- **Confirmed: the six-method `BlobPolicy` fits.** AF's `sync_files`/`cloud_only`/ |
| 923 |
`source_path` machinery maps onto the two queries + `local_path` + `reconcile` + |
| 924 |
`on_downloaded`, with `on_uploaded` correctly a no-op. GO uses 3 methods, AF uses |
| 925 |
5, the trait spans both without a special case. |
| 926 |
- **Refinement (important), `exclude` must be SQL, not a Rust closure.** AF enforces |
| 927 |
the config denylist *symmetrically*: on import (apply guard) AND on export (the |
| 928 |
changelog trigger's WHEN clause), the sensitive key must never enter the |
| 929 |
changelog in the first place (fuzz-2026-07-06 #2). A `fn(&Row)->bool` cannot |
| 930 |
compile into a generated SQL trigger, so the hook became `exclude_where: &str`, a |
| 931 |
SQL predicate the engine drops into both the trigger and the apply-time check. |
| 932 |
This is the one place the GO-derived API was actually wrong. |
| 933 |
- **Refinement: `RowIdScheme::Hashed` is per-table and adds engine duties.** Only |
| 934 |
content-bearing keys are hashed (samples/analysis/features/tags/tag_policy/ |
| 935 |
collection_members); opaque-integer and config tables stay `PrimaryKey` (M018's |
| 936 |
own split). Hashed obligates the engine to own the per-vault `row_id_salt` |
| 937 |
(generate once in sync_state, never sync), register `hash_row_id`, and emit the |
| 938 |
canonical PK into `data` on generated DELETE triggers so pull reconstructs the |
| 939 |
WHERE from the payload. Documented on the enum. |
| 940 |
- **Boundary noted: tombstone reads/sweep stay app-side.** `Tombstone{column}` |
| 941 |
owns only the apply-time write (`SET deleted_at = COALESCE(deleted_at, now())`). |
| 942 |
The `deleted_at IS NULL` read filter on domain queries and the 30-day hard-delete |
| 943 |
sweep are AF domain code, not the sync engine, the engine just stops the remote |
| 944 |
DELETE from cascading. |
| 945 |
|