Research map / Architectures and efficiency / RNNs and normalization
Learning Temporal Resolution in Spectrogram for Audio Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffRes, a differentiable drop-in module that learns frame-wise temporal resolution of mel-spectrograms by adaptively merging frames, trained jointly with an EfficientNet-B2 classifier, to cut computation while keeping accuracy on five audio tasks.
Problem
Audio classification systems use a fixed spectrogram hop size (e.g., 10 ms), but a fixed temporal resolution is not optimal across sound types and wastes computation on redundant frames.
Approach
Learn per-frame importance scores with a small 1D-CNN and adaptively warp (merge) spectrogram time frames into a lower, variable-resolution representation, plus a resolution encoding, trained end-to-end with a guide loss that pushes empty (low-energy) frames toward low importance.
Key results
- On FSD50K, the proposed method at 25 FPS reaches 56.9 mAP versus 55.6 for the 100 FPS baseline, while the best fixed-resolution alternative at 25 FPS reaches 51.4.
- With Mel+DiffRes (Best), average accuracy across AudioSet, speech recognition and music instrument reaches 92.4% versus 87.0% for the Mel baseline and 86.9% for LEAF.
- Maintains comparable FSD50K mAP (55.6) with only 15 FPS and 28% of original training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffRes to compress AudioSet tagging input to 75 FPS instead of the 100 FPS mel baseline | EfficientNet-B2 audio classifier · AudioSet tagging | mAP: 43.7\pm0.1 -> 44.2\pm0.1 | Helped |
| Use DiffRes to compress FSD50K tagging input to 25 FPS instead of the 100 FPS mel baseline | EfficientNet-B2 audio classifier · FSD50K tagging | mAP: 55.6\pm0.3 -> 56.9\pm0.2 | Helped |
| Use DiffRes to compress environmental sound input to 75 FPS instead of the 100 FPS mel baseline | EfficientNet-B2 audio classifier · Environmental sound | ACC: 85.2\pm0.5 -> 86.8\pm0.3 | Helped |
| Use DiffRes to compress SpeechCommands input to 75 FPS instead of the 100 FPS mel baseline | EfficientNet-B2 audio classifier · SpeechCommands | ACC: 97.2\pm0.1 -> 97.2\pm0.0 | No clear effect |
| Use DiffRes to compress NSynth input to 25 FPS instead of the 100 FPS mel baseline | EfficientNet-B2 audio classifier · NSynth | ACC: 79.9\pm0.2 -> 80.5\pm0.2 | Helped |
| Replace fixed hop-size reduction with DiffRes at reduced temporal resolution | EfficientNet-B2 audio classifier · Audio classification tasks in Table 1 | ACC or mAP | Helped |
| Use a 6 ms instead of 10 ms hop size | AudioSet classifier at 100 FPS · AudioSet tagging | mAP: 43.7\pm0.1 -> 44.1\pm0.1 | Helped |
| Use a 1 ms instead of 10 ms hop size | ESC-50 classifier at 100 FPS · ESC-50 | ACC: 85.2\pm0.4 -> 88.4\pm0.5 | Helped |
| Use a 3 ms instead of 10 ms hop size | SpeechCommands classifier at 100 FPS · SpeechCommands | ACC: 97.2\pm0.1 -> 97.9\pm0.1 | Helped |
| Use a 3 ms instead of 10 ms hop size | NSynth classifier at 100 FPS · NSynth | ACC: 79.9\pm0.2 -> 81.8\pm0.2 | Helped |
| Add DiffRes as the learnable front-end instead of Mel | EfficientNet-B0 audio classifier · AudioSet tagging | score (percentage): 96.8\pm0.1 -> 97.0\pm0.0 | Helped |
| Use Mel+DiffRes (Best) settings instead of Mel+DiffRes controlled settings | EfficientNet-B0 audio classifier · Speech recognition | score (percentage): 95.4\pm0.2 -> 97.9\pm0.1 | Helped |
Surprising findings
- Reducing temporal resolution to 75 FPS with DiffRes often improves over the 100 FPS baseline, which the authors attribute to a data-augmentation effect from divergent compression across training steps.
- ConvAvgPool, a learnable convolutional downsampler, does not consistently beat simple average pooling and is worse on FSD50K and environmental sound.
Limitations
- Frame importance estimator is a fairly simple 1D-CNN stack, and the authors note more sophisticated models are future work.
- Guide loss relies on an energy-based assumption that low-energy frames are unimportant, with threshold lambda introducing a tuned bias.
- Gains over fixed-resolution baselines are task-dependent; speech recognition often only matches the baseline.
Experimental setup
Datasets: AudioSet FSD50K ESC-50 SpeechCommands NSynth
Benchmarks: AudioSet FSD50K ESC-50 SpeechCommands NSynth
Metrics: mAP accuracy throughput
Baselines: Change hop size AvgPool ConvAvgPool TD-fbank SincNet LEAF DiffStride
Builds on: EfficientNet SpecAugment mixup PANNs DiffStride
Closest papers by meaning
- Audio Scanning Network: Bridging Time and Frequency Domains for Audio Classification (AAAI 2024 · similarity 0.84)
- DTF-AT: Decoupled Time-Frequency Audio Transformer for Event Classification (AAAI 2024 · similarity 0.84)
- DiffSED: Sound Event Detection with Denoising Diffusion (AAAI 2024 · similarity 0.82)
- Audio Super-Resolution with Latent Bridge Models (NeurIPS 2025 · similarity 0.81)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.81)
- PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.