Research map / Video / Video-language understanding
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Omni-RGPT, a region-level MLLM for images and videos that embeds sampled Token Mark tokens into region masks and text prompts, adds a training-only temporal region guide head, and releases the RegVID-300k region-level video instruction dataset.
Problem
Region-level understanding across images and videos lacks a unified, scalable region representation; box-as-text and RoI-feature methods scale poorly with frames and suffer temporal drift, and video region-level instruction data is scarce.
Approach
Predefined learnable Token Marks are assigned to each region prompt, embedded spatially into visual tokens and inserted as text placeholders, so the same token refers to a target consistently across frames; an auxiliary head classifies visual tokens by their Token Mark to guide region tracking without tracklets.
Key results
- Omni-RGPT 13B reaches 78.8% overall accuracy on Causal-VidQA, versus 69.4% for MotionEpic (7B).
- Omni-RGPT achieves 79.9% Q→AR on VCR, above ViP-LLaVA (78.9%) and GPT4RoI (78.6%).
- On RefCOCOg REC, Omni-RGPT reaches 95.99% val accuracy versus 88.19% for Groma and 74.08% for GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Omni-RGPT 7B instead of MotionEpic | Causal-VidQA · Causal-VidQA validation | Acc@All (%): 69.4 -> 77.5 | Helped |
| Use Omni-RGPT instead of ViP-LLaVA | VCR · VCR validation | Q → AR accuracy (%): 78.9 -> 79.9 | Helped |
| Use Omni-RGPT instead of RegionGPT | region-level video captioning · Vid-STG | METEOR: 10.4 -> 14.2 | Helped |
| Use Omni-RGPT instead of RegionGPT | region-level video captioning · Extended-Elysium | METEOR: 14.4 -> 19.3 | Helped |
| Use Omni-RGPT instead of Elysium | region-level video captioning · Elysium | METEOR: 19.1 -> 9.3 | Hurt |
| Use Omni-RGPT instead of RegionGPT | region-level video captioning · BenSMOT | METEOR: 14.0 -> 14.6 | Helped |
| Use Omni-RGPT instead of RegionGPT | region-level image captioning · RefCOCOg | CIDEr: 109.9 -> 109.7 | No clear effect |
| Use Omni-RGPT instead of RegionGPT | region-level image captioning · Visual Genome | CIDEr: 145.6 -> 139.3 | Hurt |
| Use Omni-RGPT instead of Groma | referring expression comprehension · RefCOCOg val | REC accuracy (%): 88.19 -> 95.99 | Helped |
| Add Temporal Region Guide Head (ablation) | Omni-RGPT · Extended-Elysium | METEOR: 16.2 -> 19.3 | Helped |
| Add Temporal Region Guide Head (ablation) | Omni-RGPT · BenSMOT | METEOR: 12.2 -> 14.6 | Helped |
| Reduce input from 4 frames to 3 frames (ablation) | Omni-RGPT · Causal-VidQA | Acc@All: 77.5 -> 77.2 | Hurt |
Surprising findings
- Elysium video captioning scores are lower for Omni-RGPT (9.3 METEOR) than for Elysium itself (19.1), which the authors attribute to Elysium's brief noun-style captions.
Limitations
- Uses only four uniformly sampled frames, which may not capture complex real-world or long videos.
- Dense spatial and temporal detail is needed for high-fidelity region understanding, which the current framework struggles to provide.
- Region-level video captioning underperforms Elysium on the Elysium benchmark due to caption style mismatch.
Experimental setup
Datasets: VCR Vid-STG Causal-VidQA ReVOS RegVID-300k RegionGPT region data LLaVA image-text data Elysium BenSMOT RefCOCOg Visual Genome COCO
Benchmarks: Causal-VidQA VCR Vid-STG Extended-Elysium Elysium BenSMOT RefCOCOg Visual Genome COCO
Metrics: accuracy METEOR CIDEr
Baselines: MotionEpic GPT4RoI ViP-LLaVA Groma RegionGPT Elysium GLaMM LLaVA-OV
Builds on: LLaVA RegionGPT SoM VFC
Closest papers by meaning
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.91)
- OmniViD: A Generative Framework for Universal Video Understanding (CVPR 2024 · similarity 0.89)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.88)
- RegionReasoner: Region-Grounded Multi-Round Visual Reasoning (ICLR 2026 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.87)
- VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.