Research map / Large language models / Agents, code and math
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds video-SALMONN-o1, an open-source reasoning-enhanced audio-visual LLM trained with reasoning-intensive SFT data and a step-level pairwise DPO (pDPO) with contrastive step selection, and introduces RivaBench for reasoning-intensive video understanding.
Problem
Reasoning optimization for LLMs has mostly targeted math and image inputs, leaving general audio-visual video understanding with step-by-step reasoning underexplored; existing audio-visual LLMs answer directly without reasoning.
Approach
Combines reasoning-intensive audio-visual SFT data with process DPO that uses pairwise step-level rollouts, selecting steps for pairing by their per-token KL sensitivity to a small video perturbation (contrastive step selection), avoiding an external reward model or re-ranking pipeline.
Key results
- video-SALMONN-o1 with pDPO reaches 65.6% on VideoMME vs 58.2% for LLaVA-OneVision, and 82.3% on NExT-QA vs 79.4%.
- pDPO improves over the SFT model by 4.1, 8.1 and 5.8 points on NExT-QA, RivaBench StandUp and Academic respectively.
- On RivaBench SynthDec, video-SALMONN-o1 with pDPO reaches 17.8% F1 while open-source baselines output real for almost all videos (0.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with pDPO after SFT | video-SALMONN-o1 · NExT-QA | accuracy: 78.2% -> 82.3% | Helped |
| train with pDPO after SFT | video-SALMONN-o1 · RivaBench StandUp | accuracy: 68.6% -> 76.7% | Helped |
| train with pDPO after SFT | video-SALMONN-o1 · RivaBench Academic | accuracy: 42.5% -> 48.3% | Helped |
| use inference-time reasoning rather than direct answers (ablation) | video-SALMONN-o1 after full SFT · RivaBench StandUp | accuracy: 72.3% -> 68.6% | Hurt |
| use inference-time reasoning rather than direct answers (ablation) | video-SALMONN-o1 after full SFT · Video-MME | accuracy: 63.7% -> 62.9% | Hurt |
| remove reasoning-intensive SFT data (ablation) | video-SALMONN-o1 with reasoning at inference · RivaBench StandUp | accuracy: 68.6% -> 67.5% | Hurt |
| use only the reasoning-intensive SFT subset (ablation) | video-SALMONN-o1 · Video-MME | accuracy: 62.9% -> 58.8% | Hurt |
| use pDPO instead of ORM (ablation) | video-SALMONN-o1 after SFT · Video-MME | accuracy: 62.7% -> 65.6% | Helped |
| use pDPO instead of PRM (ablation) | video-SALMONN-o1 after SFT · RivaBench StandUp | accuracy: 72.1% -> 76.7% | Helped |
| use majority voting with 20 sampled paths instead of 1-best SFT output (ablation) | video-SALMONN-o1 after SFT · NExT-QA | accuracy: 78.2% -> 81.5% | Helped |
| add intermediate-step preference pairs to full-solution pDPO (ablation) | video-SALMONN-o1 · Video understanding benchmarks | accuracy | Helped |
| perform zero-shot synthetic video detection with reasoning | video-SALMONN-o1 · RivaBench SynthDec | F1-score: 0.0% -> 17.8% | Helped |
Surprising findings
- After SFT, direct answering outperforms step-by-step reasoning on VideoMME and NExT-QA, attributed to exposure bias in teacher forcing.
- ORM and PRM trained on raw scores gave only mixed or marginal gains, while pairwise pDPO was much more effective.
Limitations
- Synthetic video detection remains low in absolute performance (17.8% F1), and most synthetic videos are still missed.
- Dependence on proprietary models (Gemini-1.5-pro, GPT-4o) for data generation and quality checks.
- Step selection assumes errors concentrate on video-dependent steps (over 70% by the authors' held-out analysis), so text-only logic errors rely on full-path preference pairs.
Experimental setup
Datasets: LibriSpeech-960h AudioCaps Video-MME NExT-QA M3AV
Benchmarks: Video-MME NExT-QA RivaBench
Metrics: accuracy F1-score precision recall
Baselines: Gemini-1.5-pro GPT-4o LLaVA-OneVision video-SALMONN Video-LLaMA 2.1 ORM PRM
Builds on: video-SALMONN 2 direct preference optimization process reward model LLaVA-OneVision Bradley-Terry model
Closest papers by meaning
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.92)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.89)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.