Research map / Video / Video segmentation and tracking
Enhancing Audiovisual Speech Recognition Through Bifocal Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BPO-AVASR, which fine-tunes the audio-only OWSM v3.1 ASR model with CLIP visual tokens via SFT, then applies bifocal DPO-style preference optimization using input-side (masked audio, flipped vision) and output-side (ChatGPT-rewritten transcripts) negatives.
Problem
Audiovisual ASR on unconstrained real-world videos suffers from noisy audio, spontaneous speech, and underuse of visual information, while prior AV-ASR models are trained only with standard SFT/ASR objectives.
Approach
Extends DPO to AV-ASR with bifocal preference pairs: input-side negatives created by manipulating audio or video inputs and output-side negatives created by rewriting ground-truth transcripts (homophone, spontaneous, vision-based errors via ChatGPT and dense video captions), optimized jointly.
Key results
- BPO-AVASR medium reaches 9.2% WER on How2, a 32.4% relative reduction over AVFormer†, and 14.5% on VisSpeech (11.6% relative improvement).
- On Ego4D, BPO-AVASR medium achieves 56.5% WER versus 64.6% for AVFormer† (12.5% relative reduction), trained on only 300 hours of How2.
- Against its SFT-only counterpart OWSM-visual small, the mixed preference dataset lowers How2 WER from 10.5 to 9.3 (11.4% relative).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply bifocal preference optimization to OWSM v3.1 | BPO-AVASR small · How2 | WER: 10.5 -> 9.3 | Helped |
| Apply bifocal preference optimization to OWSM v3.1 | BPO-AVASR small · VisSpeech | WER: 15.8 -> 15.6 | Helped |
| Apply bifocal preference optimization to OWSM v3.1 | BPO-AVASR small · Ego4D | WER: 59.9 -> 59.2 | Helped |
| Scale from small to medium model | BPO-AVASR · How2, VisSpeech, and Ego4D | WER: 9.3 -> 9.2; 15.6 -> 14.5; 59.2 -> 56.5 | Helped |
| Add masked-audio preference (ablation) | OWSM-visual small · How2, VisSpeech, and Ego4D | WER: 10.5 -> 9.4; 15.8 -> 15.7; 59.9 -> 59.2 | Helped |
| Use random-cropped vision preference instead of no preference optimization (ablation) | OWSM-visual small · How2 | WER: 10.5 -> 12.1 | Hurt |
| Use flipped-vision preference instead of no preference optimization (ablation) | OWSM-visual small · How2 | WER: 10.5 -> 9.7 | Helped |
| Add spontaneous-speech output preference (ablation) | OWSM-visual small · Ego4D | WER: 59.9 -> 59.0 | Helped |
| Mix all preference construction strategies (ablation) | BPO-AVASR small · How2, VisSpeech, and Ego4D | WER: 10.5 -> 9.3; 15.8 -> 15.6; 59.9 -> 59.2 | Helped |
| Fine-tune with bifocal preferences constructed from Ego4D training data | BPO-AVASR small · Ego4D validation | WER: 52.3 -> 50.0 | Helped |
Surprising findings
- Random cropping of visual input as a hard negative gave worse How2 WER (12.1) than the unmodified SFT baseline (10.5), suggesting ineffective negatives can hurt.
- Rule-based homophone replacement for output-side negatives improved less than LLM-generated negatives, and the AVFormer-style large-data models are not uniformly stronger, with AVATAR† best on How2 but poor on Ego4D.
Limitations
- Preference data is built from the How2 dataset and relies on ChatGPT-generated negatives, so quality and cost depend on the proxy generator.
- Evaluation is limited to three datasets, with How2 being instructional video and the others small or noisy benchmarks.
Experimental setup
Datasets: How2 Ego4D HowTo100M
Benchmarks: How2 VisSpeech Ego4D
Metrics: WER
Baselines: OWSM-ft small BEST-RQ How2 base VAT MultiRes LLD AVATAR AVFormer SynesLM
Builds on: DPO OWSM v3.1 AVFormer CLIP
Closest papers by meaning
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.89)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.85)
- MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition (ICML 2025 · similarity 0.85)
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.85)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.85)
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.