Research map / Video / Video-language understanding
WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add original audio to video (ablation) | Gemini 1.5 Pro · WorldSense | accuracy (%): 34.4 -> 48.0 | Improved |
| add subtitles to video (ablation) | Gemini 1.5 Pro · WorldSense | accuracy (%): 34.4 -> 39.3 | Improved |
| add original audio to video (ablation) | Qwen2.5-Omni · WorldSense | accuracy (%): 39.2 -> 45.2 | Improved |
| add subtitles to video (ablation) | Qwen2.5-Omni · WorldSense | accuracy (%): 39.2 -> 39.2 | No clear effect |
| add subtitles to video (ablation) | Unified-IO-2 L · WorldSense | accuracy (%): 26.6 -> 14.8 | Hurt |
| add original audio to video (ablation) | Unified-IO-2 L · WorldSense | accuracy (%): 26.6 -> 23.3 | Hurt |
| add original audio instead of subtitles (ablation) | VideoLLaMA2 · WorldSense | accuracy (%): 26.1 vs 25.4 | Mixed |
| add subtitles to video-only MLLM (ablation) | Qwen2-VL · WorldSense | accuracy (%): 32.4 -> 41.2 | Improved |
| add subtitles to video-only MLLM (ablation) | GPT 4o · WorldSense | accuracy (%): 42.6 -> 50.1 | Improved |
| add subtitles to video-only MLLM (ablation) | Video-LLaVA · WorldSense | accuracy (%): 20.3 -> 16.0 | Hurt |
| add video frames to audio-only input (ablation) | Gemini 1.5 Pro · WorldSense | accuracy (%): 34.6 -> 48.0 | Improved |
| use audio-visual MLLM instead of video-only MLLM | open-source models · WorldSense | accuracy (%) | Hurt |
Closest papers by meaning
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.92)
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos (ICLR 2025 · similarity 0.92)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
