Research map / Label-efficient and robust learning / Self-supervised and contrastive
Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MR-HuBERT, a hierarchical Transformer trained with multi-resolution HuBERT-style masked unit prediction (20ms and 40ms), which improves over HuBERT on LibriSpeech ASR, SUPERB, and ML-SUPERB while cutting inference MACs by 9-13%.
Problem
Speech SSL models process speech at a fixed 20ms frame rate, ignoring that different resolutions carry different information; prior multi-resolution work needed separate pre-trained models, and no work integrated multi-resolution information into pre-training.
Approach
A single hierarchical Transformer encodes speech at high (20ms) and low (40ms) resolutions with up/down-sampling modules between encoders, trained with summed masked unit prediction losses at both resolutions, where low-resolution targets are obtained by skip-subsampling high-resolution K-means units.
Key results
- mono-large reduces 1-hour LibriSpeech test-clean WER from 14.40 (HuBERT-large) to 6.37, a 40-50% relative reduction on 1-hour fine-tuning.
- mono-base achieves a SUPERB general score of 708.7 vs 670.4 for HuBERT-base, and mono-large reaches 864.6 vs 813.4 for HuBERT-large.
- MR-HuBERT reduces MACs from 431G to 394G in base (9%) and from 1116G to 971G in large (13%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace single-resolution HuBERT with multi-resolution HuBERT | HuBERT-large · LibriSpeech 1-hour labeled, test-clean | WER: 14.40 -> 6.37 | Helped |
| replace single-resolution HuBERT with multi-resolution HuBERT | HuBERT-large · SUPERB Understanding | score: 932.6 -> 949.7 | Helped |
| replace single-resolution HuBERT with multi-resolution HuBERT | HuBERT-large · SUPERB Enhancement | score: 456.0 -> 609.5 | Helped |
| train a multilingual multi-resolution model on Voxpopuli | HuBERT-base · ML-SUPERB 10-minute/1-hour | SUPERB score: 746.2 / 812.7 -> 957.2 / 986.8 | Helped |
| use multi-resolution inference | HuBERT-base · Inference, base model | MACs: 431G -> 394G | Helped |
| add a third resolution (20, 40, 80 ms) (ablation) | mono-base · LibriSpeech 10-hour labeled, test-clean | WER: 8.46 -> 8.84 | Hurt |
| vary encoder layer allocation to (5, 2, 5) (ablation) | mono-base · LibriSpeech 1-hour labeled, test-clean | WER: 19.26 -> 18.63 | Helped |
| simplify upsampling and downsampling modules (ablation) | mono-base · LibriSpeech 1-hour labeled, test-clean | WER: 19.26 -> 18.33 | Helped |
| use a single prediction target instead of multi-resolution prediction (ablation) | mono-base · LibriSpeech 10-hour labeled, test-clean | WER: 8.46 -> 9.52 | Hurt |
| use a single resolution (20, 20 ms) (ablation) | mono-base · LibriSpeech 100-hour labeled, test-clean | WER: 4.92 -> 4.93 | Hurt |
| reduce each encoder from four layers to three (ablation) | mono-base · LibriSpeech 100-hour labeled, test-clean | WER: 4.92 -> 5.55 | Hurt |
| use HuBERT-base-40 units for low-resolution prediction targets instead of subsampled high-resolution units (ablation) | mono-base · LibriSpeech 10-hour labeled, test-clean | WER: 8.46 -> 9.36 | Hurt |
Surprising findings
- Three-resolution MR-HuBERT does not outperform the two-resolution variant on ASR, suggesting lower resolutions do not always help ASR.
- Single-resolution MR-HuBERT with the same architecture has higher MACs (439G) than HuBERT-base, showing the efficiency gain comes from multi-resolution sequence reduction.
Limitations
- Three-resolution variants underperform the two-resolution model on ASR, and the reason is only hypothesized (per-resolution capacity).
- Some ablations restricted to the base setting and to the 10-hour ASR scenario due to suboptimal alternative units.
Experimental setup
Datasets: LibriSpeech LibriLight VoxPopuli ML-SUPERB
Benchmarks: LibriSpeech SUPERB ML-SUPERB
Metrics: word error rate phone error rate accuracy F1 CER BLEU STOI PESQ SI-SDRi SUPERB score
Baselines: HuBERT-base HuBERT-base+ HuBERT-large HuBERT-large* mHuBERT-base mHuBERT-base*
Builds on: HuBERT wav2vec 2.0 SUPERB ML-SUPERB
Closest papers by meaning
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing (NeurIPS 2024 · similarity 0.85)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.84)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.84)
- Speech Self-Supervised Learning Using Diffusion Model Synthetic Data (ICML 2024 · similarity 0.83)
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.