Research map / Vision-language and multimodal / Cross-modal learning
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
Wei-Ning Hsu, Bowen Shi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with a shared masked-cluster objective and modality dropout | u-HuBERT · LRS3 speech recognition, audio-visual, audio, and visual test inputs | WER | Improved |
| Add self-supervised pre-training | Speech translation encoder · Speech translation, average BLEU | BLEU: 3.3→35.0 | Improved |
| Enable modality dropout during pre-training (ablation) | u-HuBERT fine-tuned on audio · Speech translation, average | BLEU: 27.3→34.7 | Improved |
| Enable modality dropout during pre-training (ablation) | u-HuBERT fine-tuned on visual speech · Speech translation, average | BLEU: 17.0→31.7 | Improved |
| Remove pre-training modality dropout (ablation) | u-HuBERT · LRS3 validation, audio-visual features with video codebook | PNMI: 42.23→6.85 | Hurt |
| Fine-tune with modality dropout (ablation) | u-HuBERT pre-trained without modality dropout and fine-tuned on audio-visual speech · Speech recognition, average WER | WER: 24.5%→12.3% | Improved |
| Add audio-only unlabeled data to audio-visual pre-training (ablation) | u-HuBERT · Audio speech recognition across update counts | WER | Improved |
| Add audio-only unlabeled data to audio-visual pre-training (ablation) | u-HuBERT · Audio-visual and visual speech recognition before 400K updates | WER | Hurt |
| Add audio-only unlabeled data and train for 600K updates (ablation) | u-HuBERT · Audio-visual, audio, and visual speech recognition | WER | Improved |
| Increase visual-only input probability during pre-training modality dropout (ablation) | u-HuBERT fine-tuned on visual-only data · Speech recognition, average performance | WER | Improved |
| Increase visual-only input probability during pre-training modality dropout (ablation) | u-HuBERT fine-tuned on audio-visual or audio-only data · Speech recognition, audio-visual and audio test sets | WER | Hurt |
| Freeze all encoder layers during audio fine-tuning instead of freezing none (ablation) | u-HuBERT with pre-training modality dropout · Visual speech recognition | WER: 35.68%→30.60% | Improved |
Closest papers by meaning
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.87)
- UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation (ICLR 2025 · similarity 0.87)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)
- Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction (ICLR 2024 · similarity 0.85)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.85)
- Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
