Skip to main content

max / audiofiles

Shard the blob store one level deep on the hash prefix Blobs move from a flat store root to {root}/{ab}/{hash}.{ext}, 256 leaves keyed on the first two hex characters of the content address. The flat layout does not hold at scale. Measured over a 289k-file NSynth corpus on exFAT, import throughput fell about 90% between an empty vault and a 40,000-entry one, from 227 files/s to 21. The dedup pass isolates the cause: re-importing 5,000 already-present files writes no blob and issues no fsync, yet ran at 43.8 files/s (22.8 ms/file) into a 40k-entry vault against 227.1 files/s (4.4 ms/file) for a full copy into an empty one. Strictly less work per file, five times slower, with the corpus directory held constant. 2,294s system time against 33s user time places it in kernel filesystem metadata work, so no amount of page-cache bypassing helps; the directory has to stop being flat. One level, not git's two. Every directory occupies at least one cluster and these volumes use 128 KiB clusters, so 65,536 leaves would spend 8 GiB of directory clusters against a 36 GB library. One level costs 32 MiB and still leaves only ~1,130 blobs per leaf at 289k samples. Existing vaults keep working. sample_path now resolves the sharded location first and falls back to the legacy flat one, so every read, verify and unlink site tolerates a vault mid-migration without knowing about the layout; blob_write_path is the non-resolving write target, so new blobs always land sharded and an unmigrated vault converges as it is used. Deletes clear both layouts, and a repair write unlinks the flat copy it supersedes. The VFS mirror uses the resolving variant, since the pure mapping would link every un-relocated blob into the void. store::layout::migrate_to_sharded relocates a vault forward, with audiofiles-browser::layout_migration wrapping it off the GUI thread. The sweep is resumable, idempotent and cancellable, because on the filesystems that need it most a full pass is long enough to be interrupted, and it only records completion against a re-scan so a blob written flat mid-sweep cannot be stranded. No per-blob fsync: relocation is a rename of content-addressed data, so a crash can lose a dirent update but never bytes. Not yet wired to a trigger or progress UI; the sweep is invoked programmatically only. Also fixes the bench's blob count, which read shard directories rather than blobs and would have reported dedup as "unchanged" regardless.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-29 20:51 UTC
Signed with PGP, not checked
Commit: 9510d131197692c54c1be467a2232b4e7f44398f
Parent: 785fbeb
7 files changed, +1023 insertions, -41 deletions
M crates/audiofiles-bench/src/ingest.rs +27 -7
@@ -5,11 +5,12 @@
5 5 //! query latency that backs the browser UI.
6 6 //!
7 7 //! Two properties of the store make scale worth measuring rather than
8 - //! assuming. Blobs live in one flat directory (`store_blob_path` is
9 - //! `root.join("{hash}.{ext}")`, no fanout), so a 100k-sample vault is 100k
10 - //! entries in a single directory, and directory-lookup cost is filesystem
11 - //! dependent. And the DB runs in WAL mode with several worker connections, so
12 - //! insert cost moves with index depth.
8 + //! assuming. Blobs are sharded one level deep on the hash prefix
9 + //! (`{root}/{ab}/{hash}.{ext}`, 256 leaves), which is what a run on this bench
10 + //! bought: the flat layout it replaced lost about 90% of its import throughput
11 + //! between an empty vault and a 40,000-entry one. Directory cost is filesystem
12 + //! dependent either way, so it stays worth measuring. And the DB runs in WAL mode
13 + //! with several worker connections, so insert cost moves with index depth.
13 14 //!
14 15 //! Reported per batch rather than as one average, because the number that
15 16 //! matters is whether throughput is flat or degrading as the vault grows.
@@ -117,6 +118,25 @@
117 118 .unwrap_or(-1)
118 119 }
119 120
121 + /// Count blobs anywhere under the store root, descending into shard directories.
122 + ///
123 + /// Must recurse. Blobs live at `{root}/{ab}/{hash}.{ext}`, so counting the root's
124 + /// own entries returns the number of shard directories (at most 256) rather than
125 + /// the number of blobs, which would make the dedup check below report "unchanged"
126 + /// no matter what the store did.
127 + fn count_blobs(root: &Path) -> usize {
128 + let Ok(entries) = std::fs::read_dir(root) else {
129 + return 0;
130 + };
131 + entries
132 + .filter_map(std::result::Result::ok)
133 + .map(|e| {
134 + let path = e.path();
135 + if path.is_dir() { count_blobs(&path) } else { 1 }
136 + })
137 + .sum()
138 + }
139 +
120 140 /// Time a query, returning milliseconds. Runs it `reps` times and takes the
121 141 /// median, since a single cold query mostly measures page-cache state.
122 142 fn time_query(reps: usize, mut f: impl FnMut()) -> f64 {
@@ -409,7 +429,7 @@
409 429 println!();
410 430 println!("━━━ DEDUP (re-import of identical content) ━━━");
411 431 println!();
412 - let blobs_before = std::fs::read_dir(&samples_dir).map_or(0, std::iter::Iterator::count);
432 + let blobs_before = count_blobs(&samples_dir);
413 433 let rows_before = count_samples(&db);
414 434
415 435 let redo: Vec<&PathBuf> = files.iter().take(batch.min(files.len())).collect();
@@ -419,7 +439,7 @@
419 439 }
420 440 let redo_s = start.elapsed().as_secs_f64();
421 441
422 - let blobs_after = std::fs::read_dir(&samples_dir).map_or(0, std::iter::Iterator::count);
442 + let blobs_after = count_blobs(&samples_dir);
423 443 let rows_after = count_samples(&db);
424 444
425 445 println!(
M crates/audiofiles-browser/src/lib.rs +1
@@ -13,6 +13,7 @@
13 13 pub mod export;
14 14 pub mod import;
15 15 pub mod instrument;
16 + pub mod layout_migration;
16 17 pub mod loose_files_worker;
17 18 pub mod preview;
18 19 pub mod state;
M crates/audiofiles-core/src/config_key.rs +6
@@ -115,6 +115,12 @@
115 115 MirrorPath => "mirror_path": Local,
116 116 /// Suppresses the local import safety preflight. Local safety gate.
117 117 ImportPreflightDisabled => "import_preflight_disabled": Local,
118 + /// On-disk layout of this vault's blob directory (`flat` or `sharded`, absent
119 + /// meaning flat). Local, and load-bearingly so: it describes where THIS
120 + /// machine's bytes physically sit, and each device migrates on its own
121 + /// schedule, so replicating it would tell a peer its blobs live in a layout
122 + /// it has not moved to and make every lookup miss.
123 + BlobLayout => "blob_layout": Local,
118 124 }
119 125
120 126 impl ConfigKey {
M crates/audiofiles-core/src/vfs_mirror.rs +6 -2
@@ -14,7 +14,7 @@
14 14 use crate::SampleHash;
15 15 use crate::db::Database;
16 16 use crate::error::{Result, io_err};
17 - use crate::store::{sample_extension, sample_location, sample_source_path};
17 + use crate::store::{sample_extension, sample_location_resolved, sample_source_path};
18 18 use crate::vfs::{NodeType, list_full_tree};
19 19
20 20 /// Configuration for the mirror directory.
@@ -95,7 +95,11 @@
95 95 // samples at the store blob. Hand-rolling `store_root.join`
96 96 // here is exactly what produced dangling symlinks for
97 97 // loose-files samples (the blob never exists for them).
98 - let target = sample_location(&config.store_root, hash, ext, source).into_path();
98 + // The *resolved* variant, because a vault mid-layout-migration
99 + // still holds blobs at the legacy flat path and the pure
100 + // mapping would link every one of them into the void.
101 + let target =
102 + sample_location_resolved(&config.store_root, hash, ext, source).into_path();
99 103
100 104 // Two distinct samples can sanitize to the same display
101 105 // path in one directory. The first wins the bare name; the
M crates/audiofiles-core/src/store/mod.rs +202 -32
@@ -25,6 +25,7 @@
25 25 use crate::error::{CoreError, Result, io_err, unix_now};
26 26 use tracing::instrument;
27 27
28 + pub mod layout;
28 29 mod loose_files;
29 30 pub use loose_files::*;
30 31
@@ -112,8 +113,13 @@
112 113 }
113 114 }
114 115
115 - /// Manages on-disk sample blobs in a flat directory structure, storing files as
116 - /// `{sha256_hex}.{ext}` directly in the root directory.
116 + /// Manages on-disk sample blobs, storing files as `{sha256_hex}.{ext}` under a
117 + /// two-hex-character shard directory taken from the hash: `{root}/{ab}/{hash}.{ext}`.
118 + ///
119 + /// Vaults written before 2026-07-29 use a flat root instead. Reads resolve either
120 + /// ([`sample_path`](Self::sample_path)), writes always target the sharded layout,
121 + /// and [`layout::migrate_to_sharded`] relocates a vault forward. See [`layout`] for
122 + /// why the flat layout had to go.
117 123 ///
118 124 /// Deduplication via SHA-256: import streams the file through a hasher and skips
119 125 /// the copy if a blob with the same hash already exists.
@@ -190,12 +196,26 @@
190 196 // Mirrors the atomic temp+rename the sync download path uses. The pid
191 197 // suffix keeps two concurrent imports of the same hash from colliding on
192 198 // the temp file.
193 - let dest = self.sample_path(hash, &ext)?;
194 - let needs_write = match fs::metadata(&dest) {
195 - Ok(m) => m.len() != file_size as u64,
196 - Err(_) => true,
199 + // Writes go to the canonical sharded path, but an intact blob at the
200 + // *legacy flat* path satisfies the import just as well: a vault mid-
201 + // migration still holds blobs there, and writing a second sharded copy of
202 + // one would double the disk for no gain. So the presence check spans both
203 + // layouts while the write target stays canonical, which is what makes an
204 + // unmigrated vault converge as it is used instead of growing.
205 + let dest = self.blob_write_path(hash, &ext)?;
206 + let existing = existing_blob_path(&self.root, hash.as_str(), &ext);
207 + let needs_write = match existing.as_ref().and_then(|p| fs::metadata(p).ok()) {
208 + Some(m) => m.len() != file_size as u64,
209 + None => true,
197 210 };
198 211 if needs_write {
212 + // The shard directory is created lazily on first write into it. One
213 + // extra stat per import against a directory that almost always exists
214 + // (256 of them at most), which is far cheaper than the flat-directory
215 + // scan the sharding removes.
216 + if let Some(shard_dir) = dest.parent() {
217 + fs::create_dir_all(shard_dir).map_err(|e| io_err(shard_dir, e))?;
218 + }
199 219 let tmp = dest.with_file_name(format!("{hash}.{ext}.{}.tmp", std::process::id()));
200 220 // Copy *and* hash the bytes in a single pass, fsyncing the temp
201 221 // before the rename. `hash` was computed by an earlier pass (batch
@@ -238,6 +258,14 @@
238 258 // needs write on the directory, not the file). Best-effort: a
239 259 // filesystem that rejects the chmod must not fail the import.
240 260 set_blob_readonly(&dest);
261 + // A truncated copy at the legacy flat path has just been superseded by
262 + // the blob written at the canonical one. `sample_path` resolves sharded
263 + // first, so the flat copy is now unreachable and would leak disk
264 + // forever. Unlinking it here also means a repair write migrates a blob
265 + // as a side effect, independent of the sweep.
266 + if let Some(stale) = existing.as_ref().filter(|p| p.as_path() != dest) {
267 + let _ = fs::remove_file(stale);
268 + }
241 269 }
242 270
243 271 // Insert into DB (ignore if hash already exists). If this fails after we
@@ -268,17 +296,66 @@
268 296 Ok(self.sample_path(hash, ext)?.exists())
269 297 }
270 298
271 - /// Get the filesystem path for a sample.
299 + /// Get the filesystem path for a sample: wherever its bytes currently are.
300 + ///
301 + /// Resolves the sharded layout first, then the legacy flat one, and falls back
302 + /// to the canonical sharded path when neither exists so a caller that opens it
303 + /// surfaces a clean "not found". Resolving rather than pure is deliberate:
304 + /// every read, verify and unlink site in the codebase already goes through
305 + /// here, so a vault that has not finished migrating keeps working without each
306 + /// of those sites having to know about the layout. [`blob_write_path`](Self::blob_write_path)
307 + /// is the non-resolving write target.
272 308 ///
273 309 /// Re-validates that `hash` is exactly 64 lowercase hex characters (SHA-256)
274 310 /// as a defense-in-depth guard against a `from_trusted` slip that could form
275 311 /// a directory-traversal or malformed path.
276 312 pub fn sample_path(&self, hash: &SampleHash, ext: &str) -> Result<PathBuf> {
313 + validate_hash(hash.as_str())?;
314 + validate_extension(ext)?;
315 + Ok(existing_blob_path(&self.root, hash.as_str(), ext)
316 + .unwrap_or_else(|| store_blob_path(&self.root, hash.as_str(), ext)))
317 + }
318 +
319 + /// Where new bytes for `hash` must be written: the canonical sharded path,
320 + /// whether or not anything is there.
321 + ///
322 + /// Distinct from [`sample_path`](Self::sample_path), which resolves to the
323 + /// blob's current location. Writes always target the canonical layout so an
324 + /// unmigrated vault converges as it is used rather than accumulating more flat
325 + /// blobs. Validates as `sample_path` does.
326 + pub fn blob_write_path(&self, hash: &SampleHash, ext: &str) -> Result<PathBuf> {
277 327 validate_hash(hash.as_str())?;
278 328 validate_extension(ext)?;
279 329 Ok(store_blob_path(&self.root, hash.as_str(), ext))
280 330 }
281 331
332 + /// Unlink this blob from both layouts, ignoring absences.
333 + ///
334 + /// Every delete site uses this rather than unlinking the resolved path alone:
335 + /// resolving returns one path, so a vault that somehow holds the blob in both
336 + /// layouts (an interrupted sweep plus a repair write) would keep the other copy
337 + /// as an unreachable orphan that no GC path reclaims. Returns the first real
338 + /// error, having attempted both.
339 + fn unlink_blob_all_layouts(&self, hash: &SampleHash, ext: &str) -> Result<()> {
340 + validate_hash(hash.as_str())?;
341 + validate_extension(ext)?;
342 + let mut first_err = None;
343 + for path in [
344 + store_blob_path(&self.root, hash.as_str(), ext),
345 + legacy_flat_blob_path(&self.root, hash.as_str(), ext),
346 + ] {
347 + match fs::remove_file(&path) {
348 + Ok(()) => {}
349 + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
350 + Err(e) => first_err = first_err.or(Some(io_err(&path, e))),
351 + }
352 + }
353 + match first_err {
354 + Some(e) => Err(e),
355 + None => Ok(()),
356 + }
357 + }
358 +
282 359 /// Remove a sample from store and database. CASCADE handles VFS/tag refs.
283 360 ///
284 361 /// Deletes the file from disk first, then the DB row. If the file delete
@@ -294,14 +371,10 @@
294 371 // content-addressed and identical regardless of tombstone state, so the
295 372 // unfiltered lookup is the correct one for locating the file to unlink.
296 373 let ext = sample_extension_any(db, hash)?;
297 - let path = self.sample_path(hash, &ext)?;
298 374
299 - // File first. ENOENT is fine, the row was already pointing at nothing.
300 - match fs::remove_file(&path) {
301 - Ok(()) => {}
302 - Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
303 - Err(e) => return Err(io_err(&path, e)),
304 - }
375 + // File first, from both layouts. ENOENT is fine, the row was already
376 + // pointing at nothing.
377 + self.unlink_blob_all_layouts(hash, &ext)?;
305 378
306 379 // Then the DB row (CASCADE handles tags, vfs_nodes, etc.).
307 380 db.conn()
@@ -372,11 +445,10 @@
372 445 [hash],
373 446 )?;
374 447 if n > 0 {
375 - if let Ok(path) = self.sample_path(hash, ext)
376 - && path.exists()
377 - {
378 - let _ = fs::remove_file(&path);
379 - }
448 + // Both layouts: a vault mid-migration holds the blob at the
449 + // legacy flat path, and unlinking only the resolved one would
450 + // leave the other as an orphan this very sweep exists to stop.
451 + let _ = self.unlink_blob_all_layouts(hash, ext);
380 452 count += 1;
381 453 }
382 454 }
@@ -693,17 +765,75 @@
693 765 }
694 766 }
695 767
696 - /// Build the content-addressed store blob path for `hash`+`ext` under
697 - /// `store_root`. The single place `{store_root}/{hash}[.ext]` is constructed, no
698 - /// caller hand-rolls this join (it once drifted in the VFS mirror).
768 + /// Leading hex characters of the hash used as the blob's shard directory: one
769 + /// level of two chars, so 256 leaves.
770 + ///
771 + /// One level, not git's two. Every directory occupies at least one filesystem
772 + /// cluster, and the exFAT volumes sample libraries actually live on use 128 KiB
773 + /// clusters, so `ab/cd/` (65,536 leaves) would spend 8 GiB of directory clusters
774 + /// against a 36 GB library. One level costs 32 MiB and still holds only ~1,130
775 + /// blobs per leaf at 289k samples, small enough that a leaf stays cheap to scan
776 + /// even on a filesystem with no directory index. Measured 2026-07-29; the numbers
777 + /// and the flat-layout collapse they replace are in wiki `af-benchmarks`.
778 + pub const BLOB_SHARD_HEX: usize = 2;
779 +
780 + /// The shard directory name for `hash`: its leading [`BLOB_SHARD_HEX`] characters.
781 + ///
782 + /// A SHA-256 is uniformly distributed, so a hash prefix is a balanced shard key
783 + /// for free and the store stays purely content-addressed: nothing outside the
784 + /// content decides where a blob lives. A hash shorter than the prefix yields the
785 + /// whole string, which cannot collide with a real 64-char blob name.
786 + #[must_use]
787 + pub fn blob_shard(hash: &str) -> &str {
788 + &hash[..hash.len().min(BLOB_SHARD_HEX)]
789 + }
790 +
791 + /// Build the canonical store blob path: `{store_root}/{ab}/{hash}[.ext]`.
792 + ///
793 + /// The single place the sharded layout is constructed, no caller hand-rolls this
794 + /// join (it once drifted in the VFS mirror). See [`legacy_flat_blob_path`] for the
795 + /// layout this replaced and [`existing_blob_path`] for reads that must tolerate
796 + /// both.
699 797 pub fn store_blob_path(store_root: &Path, hash: &str, ext: &str) -> PathBuf {
798 + blob_path_in(&store_root.join(blob_shard(hash)), hash, ext)
799 + }
800 +
801 + /// Build the legacy flat blob path: `{store_root}/{hash}[.ext]`.
802 + ///
803 + /// The layout every vault created before 2026-07-29 uses. Kept as a read fallback
804 + /// so a vault that has not finished migrating still resolves, and as the source
805 + /// side of the relocation sweep in [`layout`]. Never a write target: writes go to
806 + /// [`store_blob_path`] so a vault converges as it is used.
807 + pub fn legacy_flat_blob_path(store_root: &Path, hash: &str, ext: &str) -> PathBuf {
808 + blob_path_in(store_root, hash, ext)
809 + }
810 +
811 + /// Join a blob's filename onto `dir`. Shared by both layouts so the
812 + /// extension-vs-no-extension rule cannot diverge between them.
813 + fn blob_path_in(dir: &Path, hash: &str, ext: &str) -> PathBuf {
700 814 if ext.is_empty() {
701 - store_root.join(hash)
815 + dir.join(hash)
702 816 } else {
703 - store_root.join(format!("{hash}.{ext}"))
817 + dir.join(format!("{hash}.{ext}"))
704 818 }
705 819 }
706 820
821 + /// The blob's existing on-disk path, preferring the sharded layout and falling
822 + /// back to the legacy flat one. `None` when neither exists.
823 + ///
824 + /// Sharded is checked first so a fully migrated vault costs one stat, not two.
825 + pub fn existing_blob_path(store_root: &Path, hash: &str, ext: &str) -> Option<PathBuf> {
826 + let sharded = store_blob_path(store_root, hash, ext);
827 + if sharded.exists() {
828 + return Some(sharded);
829 + }
830 + let flat = legacy_flat_blob_path(store_root, hash, ext);
831 + if flat.exists() {
832 + return Some(flat);
833 + }
834 + None
835 + }
836 +
707 837 /// Single source of truth for "where does this sample's bytes live": the
708 838 /// loose-files `source_path` if the sample has one, otherwise the store blob.
709 839 ///
@@ -724,6 +854,28 @@
724 854 }
725 855 }
726 856
857 + /// [`sample_location`] with the dual-layout existence fallback applied to the
858 + /// store case: resolves to wherever the blob currently is, sharded or legacy flat.
859 + ///
860 + /// For callers that hold a `store_root` but no [`SampleStore`] (the VFS mirror) and
861 + /// must not emit a path that does not exist. A mirror built from the pure mapping
862 + /// against a vault mid-migration would point every symlink at a sharded blob that
863 + /// has not been relocated yet, which is the dangling-symlink failure the pure
864 + /// version's own doc comment warns about. Loose samples are unaffected.
865 + pub fn sample_location_resolved(
866 + store_root: &Path,
867 + hash: &str,
868 + ext: &str,
869 + source_path: Option<&str>,
870 + ) -> SampleLocation {
871 + match sample_location(store_root, hash, ext, source_path) {
872 + SampleLocation::Store(canonical) => {
873 + SampleLocation::Store(existing_blob_path(store_root, hash, ext).unwrap_or(canonical))
874 + }
875 + loose @ SampleLocation::Loose(_) => loose,
876 + }
877 + }
878 +
727 879 /// Resolve the actual file path for a sample, checking source_path first.
728 880 ///
729 881 /// Builds on [`sample_location`] (the canonical loose-vs-store decision) and adds
@@ -893,6 +1045,28 @@
893 1045 path
894 1046 }
895 1047
1048 + /// Count every file anywhere under `root`, shard directories included.
1049 + ///
1050 + /// Blob-count assertions must not stop at the root's own entries: under the
1051 + /// sharded layout that reads 0 whatever the store actually holds, which would
1052 + /// turn an orphan-detection test into one that cannot fail.
1053 + fn count_blobs_recursively(root: &Path) -> usize {
1054 + let Ok(entries) = fs::read_dir(root) else {
1055 + return 0;
1056 + };
1057 + entries
1058 + .filter_map(std::result::Result::ok)
1059 + .map(|e| {
1060 + let path = e.path();
1061 + if path.is_dir() {
1062 + count_blobs_recursively(&path)
1063 + } else {
1064 + 1
1065 + }
1066 + })
1067 + .sum()
1068 + }
1069 +
896 1070 /// Give a sample one VFS placement, so CASCADE behaviour and placement
897 1071 /// preservation are observable.
898 1072 fn place_sample(db: &Database, hash: &str) {
@@ -1249,14 +1423,10 @@
1249 1423 assert_eq!(h1, h2, "identical bytes hash to the same sample");
1250 1424
1251 1425 // Exactly one blob on disk: the second import must reuse `{hash}.wav`, not
1252 - // write an unreachable `{hash}.aiff` orphan.
1253 - let blob_count = || {
1254 - std::fs::read_dir(store.root())
1255 - .unwrap()
1256 - .filter_map(std::result::Result::ok)
1257 - .filter(|e| e.path().is_file())
1258 - .count()
1259 - };
1426 + // write an unreachable `{hash}.aiff` orphan. Counted recursively, because
1427 + // blobs live under a shard directory now; counting only the root's own files
1428 + // would read 0 here and pass this test vacuously for the wrong reason.
1429 + let blob_count = || count_blobs_recursively(store.root());
1260 1430 assert_eq!(
1261 1431 blob_count(),
1262 1432 1,
A crates/audiofiles-browser/src/layout_migration.rs +374
@@ -1,0 +1,374 @@
1 + //! Background worker for the blob-directory layout migration.
2 + //!
3 + //! Mirrors the pattern in `cleanup.rs`: a dedicated thread with its own Database +
4 + //! SampleStore, communicating via channels, with the GUI thread polling events each
5 + //! frame. The work itself lives in `audiofiles_core::store::layout`; this is only
6 + //! the off-GUI-thread wrapper plus progress throttling.
7 + //!
8 + //! Why it must be off the GUI thread rather than a startup step: the sweep is one
9 + //! rename per blob on a filesystem whose metadata operations are the reason the
10 + //! migration exists at all. On the measured 289k-file library that is minutes at
11 + //! best, so doing it inline at vault open would present as a hang.
12 +
13 + use std::path::PathBuf;
14 +
15 + use tracing::{error, info, instrument, warn};
16 +
17 + use audiofiles_core::config_key::ConfigKey;
18 + use audiofiles_core::db::Database;
19 + use audiofiles_core::store::SampleStore;
20 + use audiofiles_core::store::layout::{self, LayoutMigration};
21 + use audiofiles_core::vfs_mirror::{MirrorConfig, sync_mirror};
22 + use audiofiles_core::worker_runtime::{WorkerCtx, WorkerHandle, spawn_worker};
23 +
24 + /// Emit at most one [`LayoutEvent::Progress`] per this many blobs handled.
25 + ///
26 + /// The event channel is bounded (4096) and applies backpressure, so an unthrottled
27 + /// per-blob emit over a 289k-blob sweep would have the worker waiting on a GUI that
28 + /// redraws at most 60 times a second, turning a progress bar into a brake. One
29 + /// event per 256 blobs is far finer than a human can perceive on a bar that takes
30 + /// minutes to fill.
31 + const PROGRESS_STRIDE: usize = 256;
32 +
33 + /// Command sent from the GUI thread to the layout worker.
34 + pub enum LayoutCommand {
35 + /// Start (or resume) relocating flat blobs into their shard directories.
36 + Migrate,
37 + /// Cancel the running migration (sets the worker's cancel flag synchronously).
38 + Cancel,
39 + }
40 +
41 + /// Event sent from the layout worker back to the GUI thread.
42 + pub enum LayoutEvent {
43 + /// Progress through the blobs enumerated at the start of this pass.
44 + Progress { completed: usize, total: usize },
45 + /// The pass finished, was cancelled, or failed to start.
46 + Complete {
47 + /// Blobs relocated into a shard.
48 + moved: usize,
49 + /// Redundant flat blobs discarded because the shard already matched.
50 + deduped: usize,
51 + /// Blobs that could not be relocated.
52 + errors: usize,
53 + /// Stopped early on request. The vault stays resolvable and resumable.
54 + cancelled: bool,
55 + /// The vault is now fully sharded and recorded as such.
56 + completed: bool,
57 + /// The VFS mirror was rebuilt after a completed migration.
58 + mirror_rebuilt: bool,
59 + },
60 + }
61 +
62 + impl LayoutEvent {
63 + /// A `Complete` reporting that nothing ran, for the failure paths that must
64 + /// still emit a terminal event so the GUI's busy flag clears.
65 + fn failed() -> Self {
66 + LayoutEvent::Complete {
67 + moved: 0,
68 + deduped: 0,
69 + errors: 1,
70 + cancelled: false,
71 + completed: false,
72 + mirror_rebuilt: false,
73 + }
74 + }
75 + }
76 +
77 + /// Handle for communicating with the background layout worker.
78 + pub struct LayoutHandle(WorkerHandle<LayoutCommand, LayoutEvent>);
79 +
80 + impl LayoutHandle {
81 + /// Poll for the next event without blocking.
82 + pub fn try_recv(&self) -> Option<LayoutEvent> {
83 + self.0.try_recv()
84 + }
85 +
86 + /// Send a command to the worker. Returns false if the worker is no longer
87 + /// alive, so callers don't treat a dropped command as accepted (and then wait
88 + /// forever for a terminal event that cannot arrive).
89 + pub fn send(&self, cmd: LayoutCommand) -> bool {
90 + if matches!(cmd, LayoutCommand::Cancel) {
91 + self.0.request_cancel();
92 + }
93 + self.0.send(cmd)
94 + }
95 + }
96 +
97 + /// Per-worker state: its own DB connection + store.
98 + struct LayoutWorker {
99 + db: Database,
100 + store: SampleStore,
101 + }
102 +
103 + /// Spawn the background layout-migration worker.
104 + #[instrument(skip_all)]
105 + pub fn spawn_layout_worker(db_path: PathBuf, store_root: PathBuf) -> std::io::Result<LayoutHandle> {
106 + let handle = spawn_worker(
107 + "layout-worker",
108 + move || -> Result<LayoutWorker, audiofiles_core::error::CoreError> {
109 + let db = Database::open(&db_path)?;
110 + let store = SampleStore::new(&store_root)?;
111 + Ok(LayoutWorker { db, store })
112 + },
113 + |e| {
114 + error!("Layout worker failed to open DB/store: {e}");
115 + LayoutEvent::failed()
116 + },
117 + |_state| LayoutEvent::failed(),
118 + layout_step,
119 + )?;
120 + Ok(LayoutHandle(handle))
121 + }
122 +
123 + #[allow(
124 + clippy::needless_pass_by_value,
125 + reason = "signature dictated by worker_runtime::spawn_worker step-fn contract (FnMut(&mut State, Cmd, &WorkerCtx))"
126 + )]
127 + fn layout_step(worker: &mut LayoutWorker, cmd: LayoutCommand, ctx: &WorkerCtx<LayoutEvent>) {
128 + // Cancel: the flag was already set synchronously by the handle, so the queued
129 + // command itself is a no-op.
130 + if matches!(cmd, LayoutCommand::Cancel) {
131 + return;
132 + }
133 + // A stale cancel from a previous pass must not abort this one.
134 + ctx.reset_cancel();
135 +
136 + let report = match layout::migrate_to_sharded(
137 + &worker.store,
138 + &worker.db,
139 + ctx.cancel_flag(),
140 + &mut |completed, total| {
141 + if completed % PROGRESS_STRIDE == 0 || completed == total {
142 + ctx.emit(LayoutEvent::Progress { completed, total });
143 + }
144 + },
145 + ) {
146 + Ok(report) => report,
147 + Err(e) => {
148 + error!("Layout migration failed: {e}");
149 + ctx.emit(LayoutEvent::failed());
150 + return;
151 + }
152 + };
153 +
154 + // The mirror's symlinks point at resolved blob paths, so every link to a
155 + // relocated blob is now stale. Rebuilding is only worth doing once the sweep is
156 + // actually complete: mid-migration the resolver still finds the un-moved blobs,
157 + // so a partial rebuild would be work thrown away on the next pass.
158 + let mirror_rebuilt = report.completed && report.moved > 0 && rebuild_mirror(worker);
159 +
160 + let LayoutMigration {
161 + moved,
162 + deduped,
163 + errors,
164 + cancelled,
165 + completed,
166 + } = report;
167 + info!(
168 + moved,
169 + deduped, errors, cancelled, completed, mirror_rebuilt, "layout migration pass finished"
170 + );
171 + ctx.emit(LayoutEvent::Complete {
172 + moved,
173 + deduped,
174 + errors,
175 + cancelled,
176 + completed,
177 + mirror_rebuilt,
178 + });
179 + }
180 +
181 + /// Rebuild the VFS mirror if one is configured. Returns whether it ran and
182 + /// succeeded.
183 + ///
184 + /// Best-effort: the migration itself has already committed, and a mirror is a
185 + /// derived convenience tree, so a failure here is logged and reported rather than
186 + /// turned into a migration failure the user would be invited to retry.
187 + fn rebuild_mirror(worker: &LayoutWorker) -> bool {
188 + let enabled = worker
189 + .db
190 + .get_config(ConfigKey::MirrorEnabled)
191 + .ok()
192 + .flatten()
193 + .is_some_and(|v| v == "true" || v == "1");
194 + if !enabled {
195 + return false;
196 + }
197 + let Some(mirror_root) = worker.db.get_config(ConfigKey::MirrorPath).ok().flatten() else {
198 + return false;
199 + };
200 + let config = MirrorConfig {
201 + mirror_root: PathBuf::from(mirror_root),
202 + store_root: worker.store.root().to_path_buf(),
203 + };
204 + match sync_mirror(&worker.db, &config) {
205 + Ok(stats) => {
206 + info!(
207 + links_created = stats.links_created,
208 + entries_removed = stats.entries_removed,
209 + "layout migration: mirror rebuilt"
210 + );
211 + true
212 + }
213 + Err(e) => {
214 + warn!("layout migration: mirror rebuild failed: {e}");
215 + false
216 + }
217 + }
218 + }
219 +
220 + /// Whether this vault has flat blobs left to relocate.
221 + ///
222 + /// Cheap (one `read_dir`) and safe to call at vault open to decide whether to
223 + /// dispatch [`LayoutCommand::Migrate`] at all. Checks the filesystem rather than
224 + /// trusting the recorded layout alone, so a vault whose sweep was interrupted
225 + /// before it could record completion still gets picked up.
226 + pub fn migration_pending(db: &Database, store_root: &std::path::Path) -> bool {
227 + if matches!(layout::recorded_layout(db), Ok(layout::BlobLayout::Sharded)) {
228 + return false;
229 + }
230 + layout::count_flat_blobs(store_root).unwrap_or(0) > 0
231 + }
232 +
233 + #[cfg(test)]
234 + mod tests {
235 + use super::*;
236 + use audiofiles_core::store::legacy_flat_blob_path;
237 +
238 + const HASH: &str = "aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899";
239 +
240 + fn poll_complete(handle: &LayoutHandle) -> LayoutEvent {
241 + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
242 + while std::time::Instant::now() < deadline {
243 + while let Some(ev) = handle.try_recv() {
244 + if matches!(ev, LayoutEvent::Complete { .. }) {
245 + return ev;
246 + }
247 + }
248 + std::thread::sleep(std::time::Duration::from_millis(5));
249 + }
250 + panic!("layout worker did not report Complete within 10s");
251 + }
252 +
253 + #[test]
254 + fn spawn_and_drop_does_not_hang() {
255 + let dir = tempfile::TempDir::new().unwrap();
256 + let db_path = dir.path().join("audiofiles.db");
257 + let store_root = dir.path().join("store");
258 + std::fs::create_dir_all(&store_root).unwrap();
259 + let _db = Database::open(&db_path).unwrap();
260 +
261 + let handle = spawn_layout_worker(db_path, store_root).unwrap();
262 + assert!(handle.try_recv().is_none());
263 + drop(handle);
264 + }
265 +
266 + #[test]
267 + fn worker_relocates_a_flat_blob() {
268 + let dir = tempfile::TempDir::new().unwrap();
269 + let db_path = dir.path().join("audiofiles.db");
270 + let store_root = dir.path().join("store");
271 + std::fs::create_dir_all(&store_root).unwrap();
272 + let db = Database::open(&db_path).unwrap();
273 + std::fs::write(legacy_flat_blob_path(&store_root, HASH, "wav"), b"bytes").unwrap();
274 +
275 + assert!(migration_pending(&db, &store_root));
276 + drop(db);
277 +
278 + let handle = spawn_layout_worker(db_path.clone(), store_root.clone()).unwrap();
279 + assert!(handle.send(LayoutCommand::Migrate));
280 +
281 + match poll_complete(&handle) {
282 + LayoutEvent::Complete {
283 + moved,
284 + errors,
285 + completed,
286 + cancelled,
287 + ..
288 + } => {
289 + assert_eq!(moved, 1);
290 + assert_eq!(errors, 0);
291 + assert!(completed);
292 + assert!(!cancelled);
293 + }
294 + LayoutEvent::Progress { .. } => unreachable!("filtered by poll_complete"),
295 + }
296 + drop(handle);
297 +
298 + assert!(store_root.join("aa").join(format!("{HASH}.wav")).is_file());
299 + assert!(!legacy_flat_blob_path(&store_root, HASH, "wav").exists());
300 +
301 + let db = Database::open(&db_path).unwrap();
302 + assert!(!migration_pending(&db, &store_root));
303 + }
304 +
305 + #[test]
306 + fn empty_store_completes_without_work() {
307 + let dir = tempfile::TempDir::new().unwrap();
308 + let db_path = dir.path().join("audiofiles.db");
309 + let store_root = dir.path().join("store");
310 + std::fs::create_dir_all(&store_root).unwrap();
311 + let db = Database::open(&db_path).unwrap();
312 + // A fresh vault has no blobs, so nothing is pending even though the layout
313 + // has never been recorded.
314 + assert!(!migration_pending(&db, &store_root));
315 + drop(db);
316 +
317 + let handle = spawn_layout_worker(db_path, store_root).unwrap();
318 + assert!(handle.send(LayoutCommand::Migrate));
319 + match poll_complete(&handle) {
320 + LayoutEvent::Complete {
321 + moved,
322 + errors,
323 + completed,
324 + mirror_rebuilt,
325 + ..
326 + } => {
327 + assert_eq!(moved, 0);
328 + assert_eq!(errors, 0);
329 + assert!(completed, "an empty root is trivially sharded");
330 + assert!(!mirror_rebuilt, "nothing moved, so no rebuild");
331 + }
332 + LayoutEvent::Progress { .. } => unreachable!("filtered by poll_complete"),
333 + }
334 + }
335 +
336 + #[test]
337 + fn cancel_before_run_reports_cancelled_and_leaves_the_blob() {
338 + let dir = tempfile::TempDir::new().unwrap();
339 + let db_path = dir.path().join("audiofiles.db");
340 + let store_root = dir.path().join("store");
341 + std::fs::create_dir_all(&store_root).unwrap();
342 + let _db = Database::open(&db_path).unwrap();
343 + std::fs::write(legacy_flat_blob_path(&store_root, HASH, "wav"), b"bytes").unwrap();
344 +
345 + let handle = spawn_layout_worker(db_path, store_root.clone()).unwrap();
346 + // Cancel sets the flag synchronously, so the Migrate queued behind it aborts
347 + // at its first check rather than running to completion.
348 + assert!(handle.send(LayoutCommand::Cancel));
349 + assert!(handle.send(LayoutCommand::Migrate));
350 +
351 + // Migrate resets the cancel flag at its start (so a stale cancel cannot
352 + // wedge every future pass), which means this run legitimately completes.
353 + // The point of the test is that the sequence terminates with a real event
354 + // and the data survives either way.
355 + match poll_complete(&handle) {
356 + LayoutEvent::Complete { errors, .. } => assert_eq!(errors, 0),
357 + LayoutEvent::Progress { .. } => unreachable!("filtered by poll_complete"),
358 + }
359 + drop(handle);
360 +
361 + let found = store_root.join("aa").join(format!("{HASH}.wav")).is_file()
362 + || legacy_flat_blob_path(&store_root, HASH, "wav").is_file();
363 + assert!(found, "the blob must exist in one layout or the other");
364 + }
365 +
366 + #[test]
367 + fn layout_event_variants_constructible() {
368 + let _ = LayoutEvent::Progress {
369 + completed: 1,
370 + total: 2,
371 + };
372 + let _ = LayoutEvent::failed();
373 + }
374 + }
A crates/audiofiles-core/src/store/layout.rs +407
@@ -1,0 +1,407 @@
1 + //! Blob-directory layout: the flat-to-sharded migration and its bookkeeping.
2 + //!
3 + //! Vaults created before 2026-07-29 keep every blob in one flat directory. That
4 + //! collapses at scale: measured on a 289k-file library, import throughput fell
5 + //! about 90% between an empty vault and a 40,000-entry one, and a *dedup* pass
6 + //! doing strictly less work per file (no blob write, no fsync) still ran five
7 + //! times slower than a full write into an empty directory. The cost is kernel
8 + //! filesystem metadata work, so it cannot be optimised away on the read side; the
9 + //! directory has to stop being flat. Numbers and method in wiki `af-benchmarks`.
10 + //!
11 + //! [`migrate_to_sharded`] relocates a vault forward. It is resumable and
12 + //! idempotent, because on the filesystems that need it most a full sweep of a
13 + //! large library takes long enough to be interrupted: every step is a rename to a
14 + //! path derived from the blob's own content, so re-running continues rather than
15 + //! repeating, and an interrupted sweep leaves a vault that still resolves (reads
16 + //! check both layouts, see [`super::existing_blob_path`]).
17 + //!
18 + //! <!-- wiki: af-benchmarks -->
19 +
20 + use std::path::Path;
21 + use std::sync::atomic::{AtomicBool, Ordering};
22 +
23 + use tracing::{instrument, warn};
24 +
25 + use super::{SampleStore, blob_shard, store_blob_path};
26 + use crate::config_key::ConfigKey;
27 + use crate::db::Database;
28 + use crate::error::{Result, io_err};
29 +
30 + /// On-disk layout of a vault's blob directory.
31 + #[derive(Debug, Clone, Copy, PartialEq, Eq)]
32 + pub enum BlobLayout {
33 + /// Every blob directly in the store root: `{root}/{hash}.{ext}`. Pre-2026-07-29.
34 + Flat,
35 + /// Blobs under a hash-prefix shard: `{root}/{ab}/{hash}.{ext}`.
36 + Sharded,
37 + }
38 +
39 + impl BlobLayout {
40 + /// The `blob_layout` config value.
41 + #[must_use]
42 + pub const fn as_str(self) -> &'static str {
43 + match self {
44 + BlobLayout::Flat => "flat",
45 + BlobLayout::Sharded => "sharded",
46 + }
47 + }
48 + }
49 +
50 + /// The layout recorded for this vault.
51 + ///
52 + /// Absent or unrecognised reads as [`BlobLayout::Flat`], which is the fail-safe
53 + /// direction: it schedules a sweep that finds nothing on an already-sharded vault
54 + /// (one `read_dir`), whereas defaulting to `Sharded` would leave a genuinely flat
55 + /// vault permanently unmigrated and paying the cost this module exists to remove.
56 + ///
57 + /// Deliberately not a schema migration. The key's only job is to hold a default,
58 + /// and a migration whose entire body inserts one default row is more moving parts
59 + /// than reading `None` as `Flat`.
60 + pub fn recorded_layout(db: &Database) -> Result<BlobLayout> {
61 + Ok(match db.get_config(ConfigKey::BlobLayout)?.as_deref() {
62 + Some(v) if v == BlobLayout::Sharded.as_str() => BlobLayout::Sharded,
63 + _ => BlobLayout::Flat,
64 + })
65 + }
66 +
67 + /// Outcome of a [`migrate_to_sharded`] pass.
68 + #[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
69 + pub struct LayoutMigration {
70 + /// Blobs relocated into a shard directory.
71 + pub moved: usize,
72 + /// Flat blobs discarded because the shard already held the same content.
73 + pub deduped: usize,
74 + /// Blobs that could not be relocated. Each is logged.
75 + pub errors: usize,
76 + /// The sweep stopped early because cancellation was requested.
77 + pub cancelled: bool,
78 + /// The root is clean and the vault is now recorded as [`BlobLayout::Sharded`].
79 + pub completed: bool,
80 + }
81 +
82 + /// Split a store-root filename into `(hash, ext)`, or `None` if it is not a blob.
83 + ///
84 + /// Strict on purpose. The store root also holds shard directories, and can hold
85 + /// `{hash}.{ext}.{pid}.tmp` leftovers from an import that died between create and
86 + /// rename. Requiring exactly one dot and a 64-char lowercase-hex stem rejects both
87 + /// a temp file (two dots) and anything a user dropped in by hand, so the sweep
88 + /// only ever renames files it is certain are blobs.
89 + fn parse_blob_name(name: &str) -> Option<(&str, &str)> {
90 + let (hash, ext) = match name.split_once('.') {
91 + Some((hash, ext)) => (hash, ext),
92 + None => (name, ""),
93 + };
94 + let is_hash = hash.len() == 64
95 + && hash
96 + .bytes()
97 + .all(|b| b.is_ascii_digit() || b.is_ascii_lowercase() && b.is_ascii_hexdigit());
98 + if !is_hash || ext.contains('.') {
99 + return None;
100 + }
101 + Some((hash, ext))
102 + }
103 +
104 + /// Every flat blob presently in the store root, as `(hash, ext)`.
105 + ///
106 + /// Non-recursive: shard directories are entries of the root and are skipped, so
107 + /// this counts exactly the work a sweep still has left.
108 + fn flat_blobs(store_root: &Path) -> Result<Vec<(String, String)>> {
109 + let mut out = Vec::new();
110 + let entries = std::fs::read_dir(store_root).map_err(|e| io_err(store_root, e))?;
111 + for entry in entries.flatten() {
112 + // A directory here is a shard (or something a user made); never a blob.
113 + if !entry.file_type().is_ok_and(|t| t.is_file()) {
114 + continue;
115 + }
116 + let name = entry.file_name();
117 + let Some(name) = name.to_str() else { continue };
118 + if let Some((hash, ext)) = parse_blob_name(name) {
119 + out.push((hash.to_string(), ext.to_string()));
120 + }
121 + }
122 + Ok(out)
123 + }
124 +
125 + /// How many flat blobs are still in the store root.
126 + ///
127 + /// Cheap enough to call before deciding whether to start a sweep: one `read_dir`.
128 + pub fn count_flat_blobs(store_root: &Path) -> Result<usize> {
129 + Ok(flat_blobs(store_root)?.len())
130 + }
131 +
132 + /// Relocate every flat blob in the store root into its hash-prefix shard.
133 + ///
134 + /// Resumable, idempotent and cancellable. `on_progress` is called with
135 + /// `(done, total)` as each blob is handled. On a clean pass (nothing cancelled, no
136 + /// errors, root verified empty of blobs afterwards) the vault is recorded as
137 + /// [`BlobLayout::Sharded`] and [`LayoutMigration::completed`] is set; otherwise the
138 + /// recorded layout is left alone so the next run picks up the remainder.
139 + ///
140 + /// No per-blob fsync. The relocation is a pure rename of content-addressed data, so
141 + /// an untimely crash can only lose the *dirent update*, never bytes: the blob is
142 + /// still at one of the two paths a read checks, and re-running the sweep finishes
143 + /// the job. Paying a metadata flush per file would reproduce the per-file fsync cost
144 + /// that the same measurement found was worth about 27% of import time.
145 + #[instrument(skip_all)]
146 + pub fn migrate_to_sharded(
147 + store: &SampleStore,
148 + db: &Database,
149 + cancel: &AtomicBool,
150 + on_progress: &mut dyn FnMut(usize, usize),
151 + ) -> Result<LayoutMigration> {
152 + let root = store.root();
153 + let pending = flat_blobs(root)?;
154 + let total = pending.len();
155 + let mut report = LayoutMigration::default();
156 +
157 + for (done, (hash, ext)) in pending.iter().enumerate() {
158 + if cancel.load(Ordering::Acquire) {
159 + report.cancelled = true;
160 + break;
161 + }
162 + let src = super::legacy_flat_blob_path(root, hash, ext);
163 + let dest = store_blob_path(root, hash, ext);
164 + let shard_dir = root.join(blob_shard(hash));
165 +
166 + if let Err(e) = std::fs::create_dir_all(&shard_dir) {
167 + warn!(shard = %shard_dir.display(), "layout: shard create failed: {e}");
168 + report.errors += 1;
169 + continue;
170 + }
171 +
172 + // A blob already at the destination is the same content by construction,
173 + // so the flat copy is redundant and should go. Size is still checked
174 + // first: if they differ, one of them is a truncated blob from a
175 + // pre-atomic-rename crash, and silently deleting either could destroy the
176 + // intact one. That case is left alone for a human and counted as an error.
177 + match (std::fs::metadata(&dest), std::fs::metadata(&src)) {
178 + (Ok(d), Ok(s)) if d.len() == s.len() => match std::fs::remove_file(&src) {
179 + Ok(()) => report.deduped += 1,
180 + Err(e) => {
181 + warn!(path = %src.display(), "layout: redundant flat blob unlink failed: {e}");
182 + report.errors += 1;
183 + }
184 + },
185 + (Ok(d), Ok(s)) => {
186 + warn!(
187 + path = %src.display(),
188 + flat_len = s.len(),
189 + sharded_len = d.len(),
190 + "layout: size mismatch between flat and sharded blob, leaving both for inspection"
191 + );
192 + report.errors += 1;
193 + }
194 + _ => match std::fs::rename(&src, &dest) {
195 + Ok(()) => report.moved += 1,
196 + Err(e) => {
197 + warn!(path = %src.display(), "layout: rename into shard failed: {e}");
198 + report.errors += 1;
199 + }
200 + },
201 + }
202 + on_progress(done + 1, total);
203 + }
204 +
205 + // One directory fsync for the whole sweep, so the renames are durable without
206 + // paying a metadata flush per blob. Best-effort: not every filesystem supports
207 + // it, and failure here only weakens durability, never correctness.
208 + if let Ok(d) = std::fs::File::open(root) {
209 + let _ = d.sync_all();
210 + }
211 +
212 + // Only claim completion against a re-scan. A blob could have been written
213 + // flat by another process between the enumeration and here, and recording
214 + // `Sharded` over one would strand it: reads would still find it via the
215 + // fallback, but nothing would ever move it.
216 + if !report.cancelled && report.errors == 0 && count_flat_blobs(root)? == 0 {
217 + db.set_config(ConfigKey::BlobLayout, BlobLayout::Sharded.as_str())?;
218 + report.completed = true;
219 + }
220 +
221 + Ok(report)
222 + }
223 +
224 + #[cfg(test)]
225 + mod tests {
226 + use super::*;
227 + use crate::SampleHash;
228 +
229 + const HASH_A: &str = "aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899";
230 + const HASH_B: &str = "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff";
231 +
232 + fn store_with_flat_blob(root: &Path, hash: &str, ext: &str, bytes: &[u8]) {
233 + std::fs::write(super::super::legacy_flat_blob_path(root, hash, ext), bytes).unwrap();
234 + }
235 +
236 + #[test]
237 + fn parse_blob_name_accepts_a_blob_with_and_without_extension() {
238 + assert_eq!(parse_blob_name(HASH_A), Some((HASH_A, "")));
239 + assert_eq!(
240 + parse_blob_name(&format!("{HASH_A}.wav")),
241 + Some((HASH_A, "wav"))
242 + );
243 + }
244 +
245 + #[test]
246 + fn parse_blob_name_rejects_temp_files_and_non_blobs() {
247 + // The exact shape import leaves behind when it dies before the rename.
248 + assert_eq!(parse_blob_name(&format!("{HASH_A}.wav.12345.tmp")), None);
249 + assert_eq!(parse_blob_name("audiofiles.db"), None);
250 + assert_eq!(parse_blob_name("notes.txt"), None);
251 + // Uppercase hex is not the spelling the store writes.
252 + assert_eq!(parse_blob_name(&HASH_A.to_uppercase()), None);
253 + // Right charset, wrong length.
254 + assert_eq!(parse_blob_name("aabbcc.wav"), None);
255 + }
256 +
257 + #[test]
258 + fn sweep_relocates_flat_blobs_and_records_the_layout() {
259 + let dir = tempfile::TempDir::new().unwrap();
260 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
261 + let root = dir.path().join("samples");
262 + let store = SampleStore::new(&root).unwrap();
263 +
264 + store_with_flat_blob(&root, HASH_A, "wav", b"aaaa");
265 + store_with_flat_blob(&root, HASH_B, "flac", b"bbbbbb");
266 +
267 + assert_eq!(recorded_layout(&db).unwrap(), BlobLayout::Flat);
268 + assert_eq!(count_flat_blobs(&root).unwrap(), 2);
269 +
270 + let cancel = AtomicBool::new(false);
271 + let mut seen = Vec::new();
272 + let report =
273 + migrate_to_sharded(&store, &db, &cancel, &mut |d, t| seen.push((d, t))).unwrap();
274 +
275 + assert_eq!(report.moved, 2);
276 + assert_eq!(report.deduped, 0);
277 + assert_eq!(report.errors, 0);
278 + assert!(report.completed);
279 + assert_eq!(seen, vec![(1, 2), (2, 2)]);
280 +
281 + // Bytes are where the sharded layout says, and gone from the flat one.
282 + assert!(store_blob_path(&root, HASH_A, "wav").exists());
283 + assert!(root.join("aa").is_dir());
284 + assert!(!super::super::legacy_flat_blob_path(&root, HASH_A, "wav").exists());
285 + assert_eq!(count_flat_blobs(&root).unwrap(), 0);
286 + assert_eq!(recorded_layout(&db).unwrap(), BlobLayout::Sharded);
287 +
288 + // And the store resolves them.
289 + let resolved = store
290 + .sample_path(&SampleHash::from_trusted(HASH_A.to_string()), "wav")
291 + .unwrap();
292 + assert_eq!(resolved, store_blob_path(&root, HASH_A, "wav"));
293 + }
294 +
295 + #[test]
296 + fn sweep_is_idempotent() {
297 + let dir = tempfile::TempDir::new().unwrap();
298 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
299 + let root = dir.path().join("samples");
300 + let store = SampleStore::new(&root).unwrap();
301 + store_with_flat_blob(&root, HASH_A, "wav", b"aaaa");
302 +
303 + let cancel = AtomicBool::new(false);
304 + let first = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
305 + let second = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
306 +
307 + assert_eq!(first.moved, 1);
308 + assert_eq!(second.moved, 0);
309 + assert!(
310 + second.completed,
311 + "a clean already-sharded vault stays sharded"
312 + );
313 + assert!(store_blob_path(&root, HASH_A, "wav").exists());
314 + }
315 +
316 + #[test]
317 + fn sweep_discards_a_redundant_flat_blob_matching_its_shard() {
318 + let dir = tempfile::TempDir::new().unwrap();
319 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
320 + let root = dir.path().join("samples");
321 + let store = SampleStore::new(&root).unwrap();
322 +
323 + // Same content in both layouts: an interrupted sweep plus a repair write.
324 + store_with_flat_blob(&root, HASH_A, "wav", b"aaaa");
325 + std::fs::create_dir_all(root.join("aa")).unwrap();
326 + std::fs::write(store_blob_path(&root, HASH_A, "wav"), b"aaaa").unwrap();
327 +
328 + let cancel = AtomicBool::new(false);
329 + let report = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
330 +
331 + assert_eq!(report.deduped, 1);
332 + assert_eq!(report.moved, 0);
333 + assert_eq!(report.errors, 0);
334 + assert!(report.completed);
335 + assert!(!super::super::legacy_flat_blob_path(&root, HASH_A, "wav").exists());
336 + }
337 +
338 + #[test]
339 + fn sweep_leaves_a_size_mismatch_alone_and_does_not_complete() {
340 + let dir = tempfile::TempDir::new().unwrap();
341 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
342 + let root = dir.path().join("samples");
343 + let store = SampleStore::new(&root).unwrap();
344 +
345 + // One of these is a truncated blob from a pre-atomic-rename crash. Deleting
346 + // either could destroy the intact copy, so both must survive the sweep.
347 + store_with_flat_blob(&root, HASH_A, "wav", b"aaaaaaaa");
348 + std::fs::create_dir_all(root.join("aa")).unwrap();
349 + std::fs::write(store_blob_path(&root, HASH_A, "wav"), b"aa").unwrap();
350 +
351 + let cancel = AtomicBool::new(false);
352 + let report = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
353 +
354 + assert_eq!(report.errors, 1);
355 + assert_eq!(report.moved, 0);
356 + assert_eq!(report.deduped, 0);
357 + assert!(
358 + !report.completed,
359 + "an unresolved blob must not record Sharded"
360 + );
361 + assert!(super::super::legacy_flat_blob_path(&root, HASH_A, "wav").exists());
362 + assert!(store_blob_path(&root, HASH_A, "wav").exists());
363 + assert_eq!(recorded_layout(&db).unwrap(), BlobLayout::Flat);
364 + }
365 +
366 + #[test]
367 + fn cancelled_sweep_keeps_the_flat_layout_recorded() {
368 + let dir = tempfile::TempDir::new().unwrap();
369 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
370 + let root = dir.path().join("samples");
371 + let store = SampleStore::new(&root).unwrap();
372 + store_with_flat_blob(&root, HASH_A, "wav", b"aaaa");
373 +
374 + let cancel = AtomicBool::new(true);
375 + let report = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
376 +
377 + assert!(report.cancelled);
378 + assert_eq!(report.moved, 0);
379 + assert!(!report.completed);
380 + assert_eq!(recorded_layout(&db).unwrap(), BlobLayout::Flat);
381 + // The blob is untouched and still resolvable, so a cancel is not data loss.
382 + assert!(super::super::legacy_flat_blob_path(&root, HASH_A, "wav").exists());
383 + }
384 +
385 + #[test]
386 + fn sweep_ignores_temp_leftovers_and_shard_directories() {
387 + let dir = tempfile::TempDir::new().unwrap();
388 + let db = Database::open(dir.path().join("audiofiles.db")).unwrap();
389 + let root = dir.path().join("samples");
390 + let store = SampleStore::new(&root).unwrap();
391 +
392 + let tmp = root.join(format!("{HASH_A}.wav.999.tmp"));
393 + std::fs::write(&tmp, b"partial").unwrap();
394 + store_with_flat_blob(&root, HASH_B, "wav", b"bbbb");
395 +
396 + let cancel = AtomicBool::new(false);
397 + let report = migrate_to_sharded(&store, &db, &cancel, &mut |_, _| {}).unwrap();
398 +
399 + assert_eq!(report.moved, 1, "only the real blob moves");
400 + assert_eq!(report.errors, 0);
401 + assert!(tmp.exists(), "a temp leftover is not the sweep's business");
402 + assert!(
403 + report.completed,
404 + "a temp file is not a flat blob, so it must not block completion"
405 + );
406 + }
407 + }