Skip to main content

max / audiofiles

Weight spectral features by frame energy instead of counting frames equally The spectral summaries were plain means over STFT frames, so a near-silent frame counted as heavily as the loudest one. A one-shot is a short loud transient followed by a long near-silent tail, and a real tail is spectrally bright (noise floor, dither, reverb) rather than digitally silent, so the tail outnumbered the sound and the mean converged on the noise floor. The `mag_sum > 0.0` guard did not help: it excludes only exact zero. Measured over 60 kick samples, centroid above the 3000 Hz gate the cymbal rule tests: 51 of 60 under the old mean, 0 of 60 energy-weighted. Where a sample has no tail at all the two agree (497 vs 632, 532 vs 593), which is what identifies the tail rather than the sample as the cause. For the centroid this is not just a nicer average. Weighting each frame by its total magnitude is algebraically identical to computing one centroid over the summed magnitude spectrum of the whole sample, which is the quantity wanted. Applied to centroid, flatness, rolloff, bandwidth and centroid variance. Deliberately NOT to onset_strength: that one is already loudness-invariant by construction (per-frame L1-normalised spectra, v3), so weighting it by frame energy would undo purpose-built behaviour, and its failure mode under a long tail is dilution toward zero rather than inflation. FEATURE_VERSION 4 -> 5. Five dims of the 35-d vector changed scale, and every consumer selects on feat_version, so stale vectors are ignored and backfilled on next launch as with v2, v3 and v4. Effect on the classification benchmark, same rules, features alone: strict accuracy 23.6% -> 33.4%, kick 15.1% -> 62.4%, percussion 20.4% -> 29.9%, cymbal over-prediction 4.35x -> 2.72x. The residual is structural: clap and tom have no rule that can emit them, so 28% of the corpus stays at zero. The regression test fails at 20,671 Hz without the weighting. No existing test caught this because the spectral tests all use synthetic sines with no tail, where the two aggregations agree.
Author: Max Johnson <me@maxj.phd> · 2026-07-29 18:22 UTC
Signed with PGP, not checked
Commit: 785fbeba1f83b6f12c1d12692a155316c0c6735a
Parent: 8addfeb
2 files changed, +117 insertions, -8 deletions
M crates/audiofiles-core/src/analysis/classify.rs +9 -1
@@ -30,7 +30,15 @@
30 30 /// low-frequency filters no longer collapse onto the same center bin (which had
31 31 /// fed duplicated energies into the DCT). The MFCC dims changed numerically;
32 32 /// backfill-on-next-launch as before.
33 - pub const FEATURE_VERSION: u32 = 4;
33 + ///
34 + /// v5: the spectral centroid, flatness, rolloff, bandwidth and centroid variance
35 + /// are now energy-weighted across frames rather than plain frame means. The old
36 + /// mean counted a near-silent frame as heavily as the loudest one, so a one-shot's
37 + /// long quiet tail (spectrally bright: noise floor, dither, reverb) dominated the
38 + /// result. Measured over 60 kicks, the plain mean put 51 above a 3000 Hz centroid
39 + /// and the weighted mean none. Five dims of the vector changed scale;
40 + /// backfill-on-next-launch as before.
41 + pub const FEATURE_VERSION: u32 = 5;
34 42
35 43 // SampleClass enum (unchanged)
36 44
M crates/audiofiles-core/src/analysis/spectral.rs +108 -7
@@ -99,6 +99,10 @@
99 99 let mut bandwidths = Vec::new();
100 100 let mut onset_diffs = Vec::new();
101 101 let mut magnitude_frames = Vec::new();
102 + // Per-frame total magnitude, used to weight the spectrum-shape averages
103 + // below. Pushed in lockstep with centroids/bandwidths/flatnesses/rolloffs,
104 + // which all only push inside the `mag_sum > 0.0` arm.
105 + let mut frame_energy = Vec::new();
102 106
103 107 // Hoisted STFT scratch, reused every frame: `windowed` is refilled before each
104 108 // FFT (realfft also uses it as scratch) and `spectrum` is overwritten by
@@ -138,6 +142,7 @@
138 142 .sum();
139 143 let frame_centroid = weighted_sum / mag_sum;
140 144 centroids.push(frame_centroid);
145 + frame_energy.push(mag_sum);
141 146
142 147 // Spectral bandwidth: standard deviation of the spectrum around the centroid.
143 148 // sqrt(Sum((freq - centroid)^2 * mag) / Sum(mag)) per frame.
@@ -230,22 +235,71 @@
230 235 }
231 236 };
232 237
233 - // Centroid variance: variance of per-frame centroids (how much the spectrum evolves).
234 - let centroid_mean = avg(&centroids);
238 + /// Energy-weighted mean of a per-frame spectral-shape feature.
239 + ///
240 + /// A plain frame mean counts a near-silent frame as heavily as the loudest
241 + /// one. That is not a small bias: a one-shot is a short loud transient
242 + /// followed by a long near-silent tail, those tail frames are spectrally
243 + /// bright (noise floor, dither, reverb), and there are far more of them
244 + /// than transient frames, so the mean converges on the noise floor rather
245 + /// than on the sound. Measured over 60 kick samples, the plain mean put 51
246 + /// of them above 3000 Hz centroid; energy-weighted, none. Where a sample
247 + /// has no tail at all the two agree, which is what identifies the tail as
248 + /// the cause.
249 + ///
250 + /// The `mag_sum > 0.0` guard on the frame loop does not help: it excludes
251 + /// only exact digital silence, and a dithered or reverberant tail is not
252 + /// exactly zero.
253 + ///
254 + /// For the centroid this is not merely a nicer average. Weighting each
255 + /// frame by its total magnitude is algebraically identical to computing one
256 + /// centroid over the summed magnitude spectrum of the whole sample, which
257 + /// is the quantity actually wanted.
258 + fn weighted_avg(values: &[f64], weights: &[f64]) -> f64 {
259 + let total: f64 = weights.iter().sum();
260 + if values.is_empty() {
261 + return 0.0;
262 + }
263 + // No energy anywhere (digital silence): fall back to the plain mean
264 + // rather than returning 0.0, which would read as a real measurement.
265 + if total <= 0.0 {
266 + return values.iter().sum::<f64>() / values.len() as f64;
267 + }
268 + values
269 + .iter()
270 + .zip(weights.iter())
271 + .map(|(v, w)| v * w)
272 + .sum::<f64>()
273 + / total
274 + }
275 +
276 + // Centroid variance: variance of per-frame centroids (how much the spectrum
277 + // evolves). Weighted like the mean it is taken around, otherwise a quiet
278 + // bright tail reads as a wildly evolving spectrum.
279 + let centroid_mean = weighted_avg(&centroids, &frame_energy);
235 280 let centroid_var = if centroids.len() < 2 {
236 281 0.0
237 282 } else {
238 - let sum_sq: f64 = centroids.iter().map(|&c| (c - centroid_mean).powi(2)).sum();
239 - sum_sq / centroids.len() as f64
283 + let sq: Vec<f64> = centroids
284 + .iter()
285 + .map(|&c| (c - centroid_mean).powi(2))
286 + .collect();
287 + weighted_avg(&sq, &frame_energy)
240 288 };
241 289
242 290 let features = SpectralFeatures {
243 291 centroid: centroid_mean,
244 - flatness: avg(&flatnesses),
245 - rolloff: avg(&rolloffs),
292 + flatness: weighted_avg(&flatnesses, &frame_energy),
293 + rolloff: weighted_avg(&rolloffs, &frame_energy),
246 294 zero_crossing_rate: zcr,
295 + // Deliberately NOT energy-weighted. Onset strength is already
296 + // loudness-invariant by construction (per-frame L1-normalised spectra,
297 + // see above), so weighting it by frame energy would undo that on
298 + // purpose-built behaviour. It is a temporal-change measure rather than
299 + // a spectral-shape one, and its failure mode under a long tail is
300 + // dilution toward zero, not inflation.
247 301 onset_strength: avg(&onset_diffs),
248 - bandwidth: avg(&bandwidths),
302 + bandwidth: weighted_avg(&bandwidths, &frame_energy),
249 303 centroid_variance: centroid_var,
250 304 };
251 305
@@ -270,6 +324,53 @@
270 324 mod tests {
271 325 use super::*;
272 326
327 + #[test]
328 + fn quiet_bright_tail_does_not_dominate_the_centroid() {
329 + // The v5 regression. A one-shot is a short loud low-frequency transient
330 + // followed by a long near-silent tail, and a real tail (noise floor,
331 + // dither, reverb) is spectrally bright rather than digitally silent.
332 + //
333 + // Built so the two aggregations disagree loudly: 4 frames of loud 200 Hz
334 + // tone, then 60 frames of very quiet high-frequency noise. A plain frame
335 + // mean lands near the noise because it is outnumbered 15:1; an
336 + // energy-weighted mean stays with the tone because that is where all the
337 + // energy is.
338 + let sr = 44100;
339 + let window = 1024;
340 + let mut samples: Vec<f32> = Vec::new();
341 + for i in 0..window * 4 {
342 + samples.push(0.9 * (2.0 * std::f32::consts::PI * 200.0 * i as f32 / sr as f32).sin());
343 + }
344 + // Deterministic pseudo-noise, alternating sign at a high rate, scaled to
345 + // roughly -80 dB. Not silence, which the `mag_sum > 0.0` guard would
346 + // have caught on its own.
347 + for i in 0..window * 60 {
348 + let s = if i % 2 == 0 { 1.0 } else { -1.0 };
349 + samples.push(s * 0.0001);
350 + }
351 +
352 + let f = compute_spectral_features(&samples, sr);
353 +
354 + // The loud content is at 200 Hz. Without weighting this test reported a
355 + // centroid up near the Nyquist-ish noise instead.
356 + assert!(
357 + f.centroid < 2000.0,
358 + "centroid {} Hz was dragged up by the quiet tail; energy-weighting is not applied",
359 + f.centroid
360 + );
361 +
362 + // And the tail must genuinely be the majority of frames, or the test is
363 + // not exercising what it claims to.
364 + let plain_mean_would_be_high = {
365 + let (_, frames) = compute_spectral_features_with_frames(&samples, sr);
366 + frames.len() > 50
367 + };
368 + assert!(
369 + plain_mean_would_be_high,
370 + "test signal did not produce enough tail frames to be a real check"
371 + );
372 + }
373 +
273 374 #[test]
274 375 #[allow(
275 376 clippy::float_cmp,