AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Wei-Ning Hsu, Bowen Shi

NeurIPS 2022 · Vision-language and multimodal › Cross-modal learning · approach: Self-supervised audio-visual representation learning · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Pre-train with a shared masked-cluster objective and modality dropoutu-HuBERT · LRS3 speech recognition, audio-visual, audio, and visual test inputsWERImproved
Add self-supervised pre-trainingSpeech translation encoder · Speech translation, average BLEUBLEU: 3.3→35.0Improved
Enable modality dropout during pre-training (ablation)u-HuBERT fine-tuned on audio · Speech translation, averageBLEU: 27.3→34.7Improved
Enable modality dropout during pre-training (ablation)u-HuBERT fine-tuned on visual speech · Speech translation, averageBLEU: 17.0→31.7Improved
Remove pre-training modality dropout (ablation)u-HuBERT · LRS3 validation, audio-visual features with video codebookPNMI: 42.23→6.85Hurt
Fine-tune with modality dropout (ablation)u-HuBERT pre-trained without modality dropout and fine-tuned on audio-visual speech · Speech recognition, average WERWER: 24.5%→12.3%Improved
Add audio-only unlabeled data to audio-visual pre-training (ablation)u-HuBERT · Audio speech recognition across update countsWERImproved
Add audio-only unlabeled data to audio-visual pre-training (ablation)u-HuBERT · Audio-visual and visual speech recognition before 400K updatesWERHurt
Add audio-only unlabeled data and train for 600K updates (ablation)u-HuBERT · Audio-visual, audio, and visual speech recognitionWERImproved
Increase visual-only input probability during pre-training modality dropout (ablation)u-HuBERT fine-tuned on visual-only data · Speech recognition, average performanceWERImproved
Increase visual-only input probability during pre-training modality dropout (ablation)u-HuBERT fine-tuned on audio-visual or audio-only data · Speech recognition, audio-visual and audio test setsWERHurt
Freeze all encoder layers during audio fine-tuning instead of freezing none (ablation)u-HuBERT with pre-training modality dropout · Visual speech recognitionWER: 35.68%→30.60%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy