Research map / Architectures and efficiency / CNN design
DTF-AT: Decoupled Time-Frequency Audio Transformer for Event Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTF-AT, a MaxViT-based audio transformer whose convolutions and convolutional stem are decoupled into time-oriented and frequency-oriented branches, evaluated on AudioSet, ESC-50 and Speech Commands V2.
Problem
Audio spectrograms have an asymmetric time-frequency information distribution, yet audio transformers typically reuse vision architectures without audio-specific layers for extracting time and frequency information.
Approach
Decouple convolutions (and the stem) into parallel time-elongated and frequency-elongated depthwise kernels, merged with a learned scalar weight, inside a MaxViT block that also uses window and grid self-attention for local and global interaction.
Key results
- Full TF variant reaches 0.473 mAP on AudioSet full set trained from scratch, a 4.4% improvement over HTS-AT (0.453).
- With ImageNet-1K initialisation, Full TF achieves 0.486 mAP on AudioSet full set, about 3.2% above the prior SOTA (0.471).
- Full TF reaches 89.2% accuracy on ESC-50 (ImageNet-1K init) and 98.3% on Speech Commands V2 (ImageNet-1K init), with ESC-50 AudioSet-pretrained at 97.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add time-frequency decoupling throughout the architecture (ablation) | DTF-AT · AudioSet full set, scratch | mAP: 0.468 -> 0.473 | Helped |
| add time-frequency decoupling throughout the architecture (ablation) | DTF-AT · ESC-50, ImageNet-1K pretrained | accuracy: 87.99 -> 89.19 | Helped |
| add time-frequency decoupling throughout the architecture (ablation) | DTF-AT · Speech Commands V2, ImageNet-1K pretrained | accuracy: 98.20 -> 98.30 | Helped |
| extend time-frequency decoupling from the stem through the full architecture (ablation) | DTF-AT · AudioSet balanced set, ImageNet-1K pretrained | mAP: 0.350 -> 0.355 | Helped |
| extend time-frequency decoupling from stage 1 to the full architecture (ablation) | DTF-AT · AudioSet full set, ImageNet-1K pretrained | mAP: 0.484 -> 0.486 | Helped |
| use kernel sizes (6×3) and (3×6) instead of (9×2) and (2×9) (ablation) | DTF-AT · AudioSet balanced set | mAP: 0.342 -> 0.352 | Helped |
| use kernel sizes (6×3) and (3×6) instead of (18×1) and (1×18) (ablation) | DTF-AT · AudioSet balanced set | mAP: 0.314 -> 0.352 | Helped |
| add TF decoupled MBConv blocks (ablation) | DTF-AT · AudioSet full set, scratch | mAP: 0.392 -> 0.450 | Helped |
| use window attention rather than replacing window attention with grid attention (ablation) | DTF-AT · AudioSet full set, scratch | mAP: 0.436 -> 0.450 | Helped |
| use grid attention rather than replacing grid attention with window attention (ablation) | DTF-AT · AudioSet full set, scratch | mAP: 0.449 -> 0.450 | Helped |
| use scalar (α, 1 - α) instead of scalar (α, β) branch weights (ablation) | DTF-AT · AudioSet balanced set | mAP: 0.349 -> 0.355 | Helped |
| use scalar (α, 1 - α) weights instead of squeeze-excitation branch weights (ablation) | DTF-AT · AudioSet balanced set | mAP: 0.346 -> 0.355 | Helped |
Surprising findings
- Full TF trained from scratch surpasses the SOTA approach initialised with ImageNet-1K weights on AudioSet full set.
- Applying decoupling in window attention instead of convolution gave lower mAP (0.443 vs 0.450).
Limitations
- Decoupling applied only to convolutional layers; window-attention decoupling not fully explored
- AudioSet subset (1.93M clips) smaller than originals, so comparisons to prior numbers are approximate
- Self-supervised pretraining not explored
Experimental setup
Datasets: AudioSet ESC-50 Speech Commands V2 ImageNet-1K
Benchmarks: AudioSet ESC-50 Speech Commands V2
Metrics: mAP accuracy
Baselines: AST PaSST HTS-AT MAST PANNs BEATs
Builds on: MaxViT AST PaSST HTS-AT PANNs
Closest papers by meaning
- M3Net: Efficient Time-Frequency Integration Network with Mirror Attention for Audio Classification on Edge (AAAI 2025 · similarity 0.86)
- Audio Scanning Network: Bridging Time and Frequency Domains for Audio Classification (AAAI 2024 · similarity 0.84)
- RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · similarity 0.84)
- Learning Temporal Resolution in Spectrogram for Audio Classification (AAAI 2024 · similarity 0.84)
- DiffSED: Sound Event Detection with Denoising Diffusion (AAAI 2024 · similarity 0.84)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.