Research map / Architectures and efficiency / Attention and transformers
Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MW-MAE, a masked autoencoder for audio whose decoder uses multi-window multi-head attention so each head attends over non-overlapping windows of different sizes, outperforming standard MAEs on ten HEAR tasks.
Problem
Standard masked autoencoders for audio use attention heads that all share the same context within a block, so local and global time-frequency information is captured only at the layer level; the paper targets better general-purpose audio representations from spectrogram MAEs.
Approach
Replace decoder multi-head attention with multi-window multi-head attention, where each head computes self-attention over non-overlapping windows of a different size (non-unary factors of the patch count plus two global heads), enabling local-global interactions within every decoder block.
Key results
- MW-MAE-L-4x16-8l reaches an overall score of 92.6 versus 90.0 for the comparable MAE-L-4x16-8l and is the best of all compared audio representations.
- MW-MAE-B-5x5-4l scores 90.6 overall versus 86.8 for MAE-B-5x5-4l; with the default base config MW-MAE scores 89.2 versus 88.1 for MAE.
- Under 90% reduction of pre-training data, MAE drops from 88.1 to 63.3 while MW-MAE drops from 89.2 to 77.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard decoder attention with multi-window attention | MAE-B-4x16-4l · HEAR 10-task overall | overall score: 88.1\ci{0.2} -> 89.2\ci{0.2} | Helped |
| replace standard decoder attention with multi-window attention | MAE-B-5x5-4l · HEAR 10-task overall | overall score: 86.8\ci{0.2} -> 90.6\ci{0.1} | Helped |
| add MW-MHA to the encoder (ablation) | MW-MAE Base (decoder only) · HEAR downstream tasks | overall score: 89.2\ci{0.2} -> 89.1\ci{0.3} | No clear effect |
| add MW-MHA to the encoder (ablation) | MW-MAE Base (decoder only) · AudioSet-20k balanced subset | linear probe mAP: 20.2\ci{0.0} -> 20.2\ci{0.0} | No clear effect |
| add MW-MHA to the encoder (ablation) | MW-MAE Base (decoder only) · AudioSet-20k balanced subset | fine-tuning mAP: 23.9\ci{0.1} -> 24.2\ci{0.1} | Helped |
| vary patch size from 8x16 to 5x5 (ablation) | MW-MAE, ViT-B · HEAR 10-task overall | overall score: 88.0±0.2 -> 90.6±0.1 | Helped |
| increase encoder size from ViT-T to ViT-L (ablation) | MW-MAE · HEAR 10-task overall | overall score: 79.0±0.3 -> 92.3±0.2 | Helped |
| increase encoder size from ViT-L to ViT-H (ablation) | MW-MAE · HEAR 10-task overall | overall score: 92.3±0.2 -> 91.1±0.2 | Hurt |
| increase decoder depth from 1 to 8 (ablation) | MW-MAE, ViT-B · HEAR 10-task overall | overall score: 87.8±0.2 -> 89.9±0.2 | Helped |
| reduce pre-training data from 100% to 10% of AS-5k (ablation) | MW-MAE, ViT-B · HEAR 10-task overall | overall score: 89.2±0.2 -> 77.2±0.3 | Hurt |
| replace standard MAE with MW-MAE under 10% AS-5k pre-training (ablation) | MAE, ViT-B · HEAR 10-task overall (10% AS-5k) | overall score: 63.3±0.2 -> 77.2±0.3 | Helped |
| replace standard attention with multi-window attention | MAE Medium Encoder · ImageNet validation | accuracy: 29.0\ci{0.0} -> 29.8\ci{0.0} | Helped |
Surprising findings
- Adding MW-MHA to the encoder gives no downstream benefit, so the gains come from decoder-only placement.
- A MW-MAE with a single decoder block performs on par with MAEs using up to 4 decoder blocks.
Limitations
- Pre-training uses AudioSet, which is orders of magnitude smaller than NLP text corpora.
- Transformer training is computationally expensive, and longer audio is a challenge.
- MW-MHA has a runtime overhead from calling attention heads individually rather than using optimized MHA kernels.
Experimental setup
Datasets: AudioSet ImageNet HEAR downstream tasks (BO, CD, ESC-50, LC, Mri-S, Mri-T, NS-5h, SC-5h, F50K, VL) AudioSet-20k balanced
Benchmarks: HEAR (10-task subset) AudioSet-20k balanced (linear probe and fine-tuning) ImageNet (linear probe proof of concept)
Metrics: normalized overall score s(m) accuracy mAP
Baselines: AudioMAE MAE-B-4x16-4l PaSST BEATs-iter3 HuBERT WavLM Wav2Vec2 SSaST HEAR-Naive
Builds on: Masked Autoencoders AudioMAE Swin-style windowed attention HEAR benchmark PWCCA
Closest papers by meaning
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.85)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.85)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.84)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.84)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.83)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.