| 1 |
# Benchmark baselines |
| 2 |
|
| 3 |
Saved `AF_BENCH_JSON` output from `audiofiles-bench`, kept so a later run has |
| 4 |
something to be compared against. Small, self-describing files: each records the |
| 5 |
metrics plus the drive, filesystem, USB link speed and RAM it was taken on. |
| 6 |
|
| 7 |
## Naming |
| 8 |
|
| 9 |
``` |
| 10 |
<mode>-<YYYY-MM-DD>-<short-sha>.json |
| 11 |
``` |
| 12 |
|
| 13 |
`mode` is the bench mode (`ingest`, `analysis`, `accuracy`, `layout`). The sha is |
| 14 |
the commit the binary was built from. Both matter: a filename alone should say what |
| 15 |
was measured and which code produced it. |
| 16 |
|
| 17 |
`layout` is the odd one out: it is a checker that happens to time a sweep, not a |
| 18 |
measurement with a baseline. It fabricates its own flat vaults, needs no corpus, |
| 19 |
and exits non-zero when a scenario fails, so it is worth running before touching |
| 20 |
anything under `store/`. Its throughput number moves with the drive rather than |
| 21 |
with the code, so it is not a figure to defend a change with. |
| 22 |
|
| 23 |
## Producing one |
| 24 |
|
| 25 |
Set `AF_BENCH_JSON` and copy the result in. The corpus lives outside the repo |
| 26 |
(gitignored, per-machine), so paths differ per box. |
| 27 |
|
| 28 |
``` |
| 29 |
AF_BENCH_CORPUS=/media/max/T9/af-corpus/samples \ |
| 30 |
AF_BENCH_VAULT=/media/max/T9/af-bench-fresh \ |
| 31 |
AF_BENCH_BATCH=400 AF_BENCH_ANALYZE=600 \ |
| 32 |
AF_BENCH_JSON=/tmp/run.json \ |
| 33 |
cargo run --release -p audiofiles-bench -- ingest |
| 34 |
``` |
| 35 |
|
| 36 |
Diffing two runs: |
| 37 |
|
| 38 |
``` |
| 39 |
diff <(jq -S . benchmarks/ingest-2026-07-29-1258f9e.json) <(jq -S . /tmp/run.json) |
| 40 |
``` |
| 41 |
|
| 42 |
## What makes a run comparable |
| 43 |
|
| 44 |
Only against another run on the same machine, the same drive, and the **same |
| 45 |
corpus contents**. All three are part of the measurement, and only the first two |
| 46 |
are recorded in the file. |
| 47 |
|
| 48 |
The corpus one has already bitten. The 2026-07-29 baselines here read 2,241 files |
| 49 |
where the earlier figure in wiki `af-benchmarks` read 1,761, because the four |
| 50 |
120-file format arms added for the per-format decode section landed inside |
| 51 |
`samples/` and the ingest walk picks them up. Aggregate throughput came out 168 |
| 52 |
files/s against the older 204, which looks like a 17% regression and is not one: |
| 53 |
it is a different set of files, weighted differently. An A/B of the same corpus |
| 54 |
across the two commits showed the change cost nothing (see below). |
| 55 |
|
| 56 |
Rules that follow from that: |
| 57 |
|
| 58 |
- **Take a baseline with the machine and drive idle.** Import is I/O bound. A run |
| 59 |
taken while a dataset was downloading to the same drive read 40.7 files/s |
| 60 |
against 204 for the same corpus, with no code change. |
| 61 |
- **Use a fresh vault path.** Pointing at an existing scratch vault measures |
| 62 |
dedup, not import, because the store finds the blobs already there. |
| 63 |
- **Do not compare across corpora.** If the corpus changed, re-measure both arms |
| 64 |
rather than reasoning about the delta. |
| 65 |
- **A/B across commits by rebuilding, not by trusting an old file.** Checking out |
| 66 |
the older commit and running it back to back on the same corpus is a few |
| 67 |
minutes and removes every confound at once. |
| 68 |
|
| 69 |
## Current files |
| 70 |
|
| 71 |
|
| 72 |
|
| 73 |
| `ingest-2026-07-29-785fbeb.json` | Pre-fanout control: flat blob store root. | |
| 74 |
| `ingest-2026-07-29-1258f9e.json` | Was the baseline. Now a pre-`1a0edd0` control. | |
| 75 |
|
| 76 |
Taken back to back, machine and drive idle, identical corpus and batch size, a |
| 77 |
fresh vault each. They come out 168.3 against 168.4 files/s and 106.5 MB/s each, |
| 78 |
so sharding the blob directory costs nothing measurable at this size. That is the |
| 79 |
expected result rather than a disappointing one: at 2,241 blobs a flat directory |
| 80 |
was never the problem. The layout change is for the 289k-file case, where the flat |
| 81 |
root lost about 90% of its throughput. |
| 82 |
|
| 83 |
**Neither file is a comparison target any more, and there is no ingest baseline |
| 84 |
until someone takes one.** `1a0edd0` changed the import write path underneath |
| 85 |
them: the per-file directory fsync became one per shard directory at the end of |
| 86 |
the run, and the streaming hash/copy buffers went from 8 KiB to 256 KiB. Both |
| 87 |
make import faster, so a run that beats 168.4 files/s says nothing about the code |
| 88 |
it is testing until it has a post-`1a0edd0` baseline to sit against. Same trap as |
| 89 |
the corpus one above, from the other direction: there, the file set moved under a |
| 90 |
fixed measurement; here, the measured code moved under a fixed file set. Take the |
| 91 |
new baseline the way the rules above say, machine and drive idle, and this section |
| 92 |
can be rewritten around it. |
| 93 |
|