Research map / Video / Video-language understanding
Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Omni-R1 trains Qwen2.5-Omni-7B as a global reasoning system with GRPO and hierarchical rewards, collaborating with a detail understanding system (grounding model plus SAM2) for referring audio-visual and reasoning video segmentation.
Problem
Omnimodal models face a trade-off between dense temporal coverage (low resolution, many frames) and fine-grained pixel grounding (high resolution), and keyframe selection and task reformulation are hard to supervise with SFT data.
Approach
A two-system design in which an omni model (System 1) selects keyframes and rewrites the task into local queries for a high-resolution grounding system (System 2 with SAM2), trained end-to-end with GRPO using hierarchical rewards (keyframe quality, frame-instruction alignment via Hungarian matching, and global mask IoU) obtained from online collaboration with System 2.
Key results
- On Ref-AVSBench, Omni-R1-7B reaches 74.2 J&F on the unseen set versus 62.3 for the base Qwen2.5-Omni-7B, and 47.2 on the seen set versus 31.6.
- On ReVOS, Omni-R1-11B (Omni-R1-8B/Sa2VA-based System 2 variant) reaches 58.9 overall J&F, above Sa2VA-26B at 58.4.
- Omni-R1-7B improves OmniBench from 47.3 to 49.3, VideoMME short from 69.8 to 73.0, and MVBench from 66.1 to 70.3 over Qwen2.5-Omni-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Omni-R1 reinforcement learning | Qwen2.5-Omni-7B · Ref-AVSBench Seen and Unseen | J&F: 31.6 -> 47.2; 62.3 -> 74.2 | Helped |
| Train with Omni-R1 reinforcement learning | Qwen2.5-Omni-7B · ReVOS Overall | J&F: 36.6 -> 47.6 | Helped |
| Use Omni-R1-7B as System 1 and Sa2VA as System 2 | Omni-R1 · ReVOS Overall | J&F: 58.4 -> 58.9 | Helped |
| Train with Omni-R1 | Qwen2.5-Omni-7B · OmniBench | Avg: 47.3 -> 49.3 | Helped |
| Add frame-instruction alignment reward R_A to R_K + R_G (ablation) | Omni-R1 · ReVOS Overall | J&F: 39.9 -> 38.4 | Hurt |
| Add global temporal consistency reward R_G to R_K (ablation) | Omni-R1 · ReVOS Overall | J&F: 35.2 -> 39.9 | Helped |
| Add 2,000 refCOCOg grounding samples for System 2 training (ablation) | Omni-R1 · ReVOS Overall | J&F: 44.7 | Helped |
| Train on 1,600 AVS samples (ablation) | Qwen2.5-Omni-7B · AVHBench JUDGE | accuracy: 58.5% -> 60.8% | Helped |
| Reduce GRPO KL coefficient beta to 0.001 (ablation) | AVS 1600-sample model · AVHBench JUDGE | accuracy: 60.8% -> 61.5% | Helped |
| Increase AVS training data to 10,400 samples (ablation) | Qwen2.5-Omni-7B · AVHBench JUDGE | accuracy: 58.5% -> 58.9% | No clear effect |
| Train on VOS tasks (ablation) | Qwen2.5-Omni-7B · AVHBench JUDGE | accuracy: 58.5% -> 66.0% | Helped |
| Jointly train on AVS and VOS tasks (ablation) | Qwen2.5-Omni-7B · AVHBench JUDGE | accuracy: 58.5% -> 71.9% | Helped |
Surprising findings
- Adding the frame-instruction alignment reward (R_A) lowered ReVOS overall score (38.4 vs 39.9 without it), which the authors attribute to instability.
- Supervised fine-tuning on RefAVS reduced seen-set performance relative to the base model (39.1 vs 31.6 is an increase, but RL is reported as improving out-of-domain generalization where SFT weakens it).
Limitations
- System 2 lacks temporal context, which may hurt temporally coherent tasks.
- One-way coarse-to-fine decoupling struggles with tasks needing finer temporal sensitivity such as anomaly localization.
- Severe cross-modal hallucination observed on RefAVS with full audio and multi-frame video, leading to a RefAID simplification.
Experimental setup
Datasets: RefAVS ReVOS MeViS RefCOCOg AVHBench
Benchmarks: Ref-AVSBench ReVOS MeViS OmniBench VideoMME MVBench AVHBench
Metrics: J&F accuracy
Baselines: Qwen2.5-Omni-7B Qwen2.5-Omni-7B (SFT) AVSBench AVSegFormer GAVS ReferFormer R2VOS EEMC Sa2VA-8B Sa2VA-26B VideoChat-R1 Video-R1
Builds on: GRPO DeepSeek-R1 SAM2 Sa2VA SegZero
Closest papers by meaning
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.90)
- OneThinker: All-in-one Reasoning Model for Image and Video (CVPR 2026 · similarity 0.89)
- LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning (NeurIPS 2026 · similarity 0.89)
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.88)
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.