| 1 |
# SyncKit Client SDK: Integration Patterns |
| 2 |
|
| 3 |
How GoingsOn, Balanced Breakfast, and audiofiles consume the SyncKit client SDK. Use this guide to add sync to a new app. |
| 4 |
|
| 5 |
## Common Architecture |
| 6 |
|
| 7 |
All three apps follow the same pattern: |
| 8 |
|
| 9 |
``` |
| 10 |
App (SQLite) |
| 11 |
├── Syncable tables with INSERT/UPDATE/DELETE triggers |
| 12 |
├── sync_changelog table (captures local mutations) |
| 13 |
├── sync_state table (key-value: device_id, cursor, flags) |
| 14 |
├── Sync service module (push/pull logic) |
| 15 |
└── Scheduler (background timer, exponential backoff) |
| 16 |
``` |
| 17 |
|
| 18 |
### Trigger-Based Changelog |
| 19 |
|
| 20 |
Each syncable table has SQL triggers that write to `sync_changelog`: |
| 21 |
- Triggers fire on INSERT, UPDATE, DELETE |
| 22 |
- When `applying_remote = '1'` in sync_state, triggers are suppressed (prevents echo-back) |
| 23 |
- Only whitelisted columns are included in the JSON payload |
| 24 |
- Row IDs are UUIDs (GO, BB) or content hashes (AF) |
| 25 |
|
| 26 |
### Push/Pull Cycle |
| 27 |
|
| 28 |
1. **Push**: Read unpushed changelog entries (batch 500) → encrypt via SDK → send to MNW → mark as pushed |
| 29 |
2. **Pull**: Fetch remote changes from MNW → decrypt via SDK → apply locally with triggers suppressed → update cursor |
| 30 |
3. **Cleanup**: Delete pushed entries older than 7 days |
| 31 |
|
| 32 |
### FK-Safe Ordering |
| 33 |
|
| 34 |
Both push and pull use foreign-key-safe ordering: |
| 35 |
- **Upserts**: Parents first (projects before tasks, VFS before nodes) |
| 36 |
- **Deletes**: Children first (tasks before projects, nodes before VFS) |
| 37 |
|
| 38 |
--- |
| 39 |
|
| 40 |
## Per-App Integration |
| 41 |
|
| 42 |
### GoingsOn (13 tables) |
| 43 |
|
| 44 |
**Tables synced:** |
| 45 |
``` |
| 46 |
projects → milestones, tasks, events |
| 47 |
tasks → annotations, subtasks |
| 48 |
contacts → contact_emails, contact_phones, contact_social_handles, contact_custom_fields |
| 49 |
email_accounts (16 config columns only — credentials excluded) |
| 50 |
``` |
| 51 |
|
| 52 |
**Special handling:** |
| 53 |
- Email account passwords and OAuth tokens are excluded from the column whitelist, never leave the device |
| 54 |
- Tasks with `source_email_id` referencing unsynced emails: FK enforcement relaxed during remote apply |
| 55 |
|
| 56 |
**Location:** `src-tauri/src/sync_service.rs` (1814 lines, 43 tests) |
| 57 |
|
| 58 |
### Balanced Breakfast (5 tables) |
| 59 |
|
| 60 |
**Tables synced:** |
| 61 |
``` |
| 62 |
feeds → feed_tags, query_feeds |
| 63 |
user_config |
| 64 |
feed_items (partial: is_read + is_starred only) |
| 65 |
``` |
| 66 |
|
| 67 |
**Special handling:** |
| 68 |
- `feed_items`: Uses UPDATE (not INSERT OR REPLACE), only syncs user read/star state, never full content |
| 69 |
- `feed_items` deletes are ignored, content re-fetches from source feeds |
| 70 |
- Changelog retention cap: MAX_CHANGELOG_ENTRIES = 10,000 (prevents unbounded growth) |
| 71 |
|
| 72 |
**Location:** `src-tauri/src/sync_service.rs` (1062 lines, 30 tests) |
| 73 |
|
| 74 |
### audiofiles (9 tables) |
| 75 |
|
| 76 |
**Tables synced:** |
| 77 |
``` |
| 78 |
vfs → vfs_nodes |
| 79 |
samples → audio_analysis, tags, collection_members |
| 80 |
collections → collection_members |
| 81 |
smart_folders |
| 82 |
user_config |
| 83 |
``` |
| 84 |
|
| 85 |
**Special handling (blob sync):** |
| 86 |
1. VFS entries have a `sync_files` flag controlling blob sync |
| 87 |
2. Samples marked `cloud_only = 1` if blob doesn't exist locally |
| 88 |
3. After push/pull, upload pending blobs (local files in sync-enabled VFS) |
| 89 |
4. Download missing blobs (cloud_only samples where file is needed) |
| 90 |
|
| 91 |
**Location:** `crates/audiofiles-sync/src/service.rs` (1438 lines, 48 tests) |
| 92 |
|
| 93 |
--- |
| 94 |
|
| 95 |
## SDK Public API |
| 96 |
|
| 97 |
### Authentication |
| 98 |
|
| 99 |
```rust |
| 100 |
use synckit_client::SyncKitClient; |
| 101 |
|
| 102 |
// Create client |
| 103 |
let client = SyncKitClient::new(SyncKitConfig { |
| 104 |
server_url: "https://makenot.work".into(), |
| 105 |
api_key: "your-app-api-key".into(), |
| 106 |
}); |
| 107 |
|
| 108 |
// OAuth2 PKCE flow |
| 109 |
let auth_url = client.build_authorize_url(port, state, code_challenge); |
| 110 |
// ... user completes browser flow ... |
| 111 |
let (user_id, app_id) = client.authenticate_with_code(code, code_verifier, port, sdk_key).await?; |
| 112 |
|
| 113 |
// Session management |
| 114 |
client.is_token_expired() -> bool |
| 115 |
client.session_info() -> Option<SessionInfo> |
| 116 |
client.clear_session() -> Result<()> |
| 117 |
``` |
| 118 |
|
| 119 |
### Encryption Setup |
| 120 |
|
| 121 |
```rust |
| 122 |
// First device: generate new master key |
| 123 |
client.setup_encryption_new(password).await?; |
| 124 |
|
| 125 |
// Subsequent devices: decrypt existing key from server |
| 126 |
client.setup_encryption_existing(password).await?; |
| 127 |
|
| 128 |
// Try restore from OS keychain (macOS Keychain, Linux secret-service, Windows Credential Manager) |
| 129 |
client.try_load_key_from_keychain().await? -> bool |
| 130 |
|
| 131 |
// Check if server has encrypted key (determines new vs existing flow) |
| 132 |
client.has_server_key().await? -> bool |
| 133 |
client.has_master_key() -> bool |
| 134 |
``` |
| 135 |
|
| 136 |
### Device Management |
| 137 |
|
| 138 |
```rust |
| 139 |
client.register_device(hostname, platform).await? -> Device |
| 140 |
client.list_devices().await? -> Vec<Device> |
| 141 |
``` |
| 142 |
|
| 143 |
### Push/Pull |
| 144 |
|
| 145 |
```rust |
| 146 |
use synckit_client::{ChangeEntry, ChangeOp}; |
| 147 |
|
| 148 |
// Push encrypted changes to server |
| 149 |
client.push(device_id, changes: Vec<ChangeEntry>).await?; |
| 150 |
|
| 151 |
// Pull decrypted changes from server |
| 152 |
let (changes, new_cursor, has_more) = client.pull(device_id, cursor).await?; |
| 153 |
``` |
| 154 |
|
| 155 |
`ChangeEntry` fields: |
| 156 |
- `table`: Table name (string) |
| 157 |
- `op`: `ChangeOp::Insert`, `Update`, or `Delete` |
| 158 |
- `row_id`: Primary key (string) |
| 159 |
- `timestamp`: When the change was made |
| 160 |
- `data`: `Option<serde_json::Value>` (None for deletes) |
| 161 |
|
| 162 |
### Blob Operations (audiofiles only) |
| 163 |
|
| 164 |
```rust |
| 165 |
// Upload a file-backed blob. Streams it a part at a time, so peak memory is |
| 166 |
// one part rather than the whole file, and dedups server-side before reading. |
| 167 |
if client.blob_upload_streaming(hash, &path).await? == BlobUploadOutcome::Uploaded { |
| 168 |
client.blob_confirm(hash, size).await?; |
| 169 |
} |
| 170 |
|
| 171 |
// Download |
| 172 |
let url = client.blob_download_url(hash).await?; |
| 173 |
let data = client.blob_download(url).await?; |
| 174 |
``` |
| 175 |
|
| 176 |
For bytes already in memory (no file to stream), the one-shot pair still exists: |
| 177 |
`blob_upload_url(hash, plaintext_size)` then `blob_upload(hash, url, data)`, then |
| 178 |
`blob_confirm`. It buffers the whole ciphertext and the server refuses it above |
| 179 |
its one-shot ceiling, so prefer the streaming call whenever there is a path. |
| 180 |
|
| 181 |
--- |
| 182 |
|
| 183 |
## First-Run Sequence |
| 184 |
|
| 185 |
### First Device |
| 186 |
|
| 187 |
1. User clicks "Connect to Sync" |
| 188 |
2. App builds auth URL with PKCE challenge → opens browser |
| 189 |
3. User logs in on MNW, approves scopes |
| 190 |
4. Browser redirects to `localhost:PORT` with authorization code |
| 191 |
5. App exchanges code for JWT via SDK |
| 192 |
6. App detects no server key → shows "Set Encryption Password" dialog |
| 193 |
7. User enters password → `setup_encryption_new(password)` generates and uploads encrypted key |
| 194 |
8. App registers device → stores device_id in sync_state |
| 195 |
9. App creates initial snapshot (INSERT all existing rows to changelog) |
| 196 |
10. First sync cycle runs |
| 197 |
|
| 198 |
### Same Device, Later Run |
| 199 |
|
| 200 |
1. App calls `try_load_key_from_keychain()` → restores session + key |
| 201 |
2. If success, ready to sync |
| 202 |
3. If keychain empty, re-run OAuth flow |
| 203 |
|
| 204 |
### Additional Device |
| 205 |
|
| 206 |
1. OAuth flow as above |
| 207 |
2. App detects server has key → shows "Enter Encryption Password" dialog |
| 208 |
3. `setup_encryption_existing(password)` decrypts server key → saves to local keychain |
| 209 |
4. Register new device, create initial snapshot, sync |
| 210 |
|
| 211 |
--- |
| 212 |
|
| 213 |
## Adding Sync to a New App |
| 214 |
|
| 215 |
### 1. Database Schema |
| 216 |
|
| 217 |
Add these tables: |
| 218 |
|
| 219 |
```sql |
| 220 |
CREATE TABLE sync_changelog ( |
| 221 |
id INTEGER PRIMARY KEY AUTOINCREMENT, |
| 222 |
table_name TEXT NOT NULL, |
| 223 |
op TEXT NOT NULL, -- 'INSERT', 'UPDATE', 'DELETE' |
| 224 |
row_id TEXT NOT NULL, |
| 225 |
timestamp INTEGER NOT NULL, |
| 226 |
data TEXT, -- JSON, NULL for DELETE |
| 227 |
pushed INTEGER DEFAULT 0 |
| 228 |
); |
| 229 |
|
| 230 |
CREATE TABLE sync_state ( |
| 231 |
key TEXT PRIMARY KEY, |
| 232 |
value TEXT NOT NULL |
| 233 |
); |
| 234 |
|
| 235 |
-- Seed defaults |
| 236 |
INSERT INTO sync_state VALUES ('pull_cursor', '0'); |
| 237 |
INSERT INTO sync_state VALUES ('applying_remote', '0'); |
| 238 |
INSERT INTO sync_state VALUES ('initial_snapshot_done', '0'); |
| 239 |
INSERT INTO sync_state VALUES ('auto_sync_enabled', '1'); |
| 240 |
INSERT INTO sync_state VALUES ('sync_interval_minutes', '15'); |
| 241 |
``` |
| 242 |
|
| 243 |
### 2. Triggers |
| 244 |
|
| 245 |
For each syncable table, add three triggers: |
| 246 |
|
| 247 |
```sql |
| 248 |
CREATE TRIGGER after_insert_my_table AFTER INSERT ON my_table |
| 249 |
BEGIN |
| 250 |
SELECT CASE |
| 251 |
WHEN (SELECT value FROM sync_state WHERE key = 'applying_remote') != '1' |
| 252 |
THEN (INSERT INTO sync_changelog (table_name, op, row_id, timestamp, data) |
| 253 |
VALUES ('my_table', 'INSERT', NEW.id, strftime('%s','now'), |
| 254 |
json_object('col1', NEW.col1, 'col2', NEW.col2))) |
| 255 |
END; |
| 256 |
END; |
| 257 |
-- Repeat for UPDATE and DELETE (DELETE uses OLD.id, data = NULL) |
| 258 |
``` |
| 259 |
|
| 260 |
### 3. Core Module |
| 261 |
|
| 262 |
Implement these functions: |
| 263 |
|
| 264 |
|
| 265 |
|
| 266 |
| `get_sync_state(key)` | Read from sync_state | |
| 267 |
| `set_sync_state(key, value)` | Write to sync_state | |
| 268 |
| `ensure_device_registered(client)` | Cache device_id after first registration | |
| 269 |
| `create_initial_snapshot()` | One-time: INSERT all existing rows to changelog | |
| 270 |
| `push_changes(client, device_id)` | Batch 500, read/encrypt/send/mark-pushed | |
| 271 |
| `pull_changes(client, device_id)` | Loop until no more, decrypt/apply/save-cursor | |
| 272 |
| `apply_upsert(table, row_id, data)` | INSERT OR REPLACE with FK order | |
| 273 |
| `apply_delete(table, row_id)` | DELETE with FK order | |
| 274 |
| `cleanup_changelog()` | Prune pushed entries older than 7 days | |
| 275 |
|
| 276 |
### 4. Scheduler |
| 277 |
|
| 278 |
- Check every 60 seconds if sync is due |
| 279 |
- On first run: `create_initial_snapshot()` if needed |
| 280 |
- Exponential backoff on failure (2^N minutes, capped at 15) |
| 281 |
- Emit status events for UI updates |
| 282 |
|
| 283 |
### 5. Commands / UI |
| 284 |
|
| 285 |
Expose these to the frontend: |
| 286 |
- `sync_status()`: configured, authenticated, encryption, device, pending changes |
| 287 |
- `sync_start_auth()`: returns auth URL + state + verifier |
| 288 |
- `sync_complete_auth(code, state, verifier)`: exchanges for JWT |
| 289 |
- `sync_setup_encryption_new/existing(password)`: calls SDK method |
| 290 |
- `sync_now()`: triggers immediate cycle |
| 291 |
- `sync_disconnect()`: clears credentials and session |
| 292 |
|
| 293 |
--- |
| 294 |
|
| 295 |
## Key Files |
| 296 |
|
| 297 |
|
| 298 |
|
| 299 |
| SDK source | `MNW/shared/synckit-client/src/` | |
| 300 |
| SDK auth | `MNW/shared/synckit-client/src/client/auth.rs` | |
| 301 |
| SDK push/pull | `MNW/shared/synckit-client/src/client/sync.rs` | |
| 302 |
| SDK encryption | `MNW/shared/synckit-client/src/crypto.rs` | |
| 303 |
| GO sync service | `Apps/goingson/src-tauri/src/sync_service.rs` | |
| 304 |
| BB sync service | `Apps/balanced_breakfast/src-tauri/src/sync_service.rs` | |
| 305 |
| AF sync service | `Apps/audiofiles/crates/audiofiles-sync/src/service.rs` | |
| 306 |
| Server endpoints | `MNW/server/src/routes/synckit.rs` | |
| 307 |
| Server DB | `MNW/server/src/db/synckit.rs` | |
| 308 |
|