Research map / Video / Video-language understanding
MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MME-VideoOCR, a manually annotated benchmark of 1,464 videos and 2,000 QA pairs across 25 video OCR tasks in 10 categories, and evaluates 18 MLLMs, with the best (Gemini-2.5 Pro) reaching 73.7%.
Problem
Existing OCR benchmarks for MLLMs focus on static images or narrow video settings and emphasize perception over text-based understanding and reasoning in video, so video OCR capability of MLLMs is poorly measured.
Approach
A comprehensive video OCR benchmark spanning perception, understanding, and reasoning with fully manual QA annotation, a debiasing test against textual priors and knowledge leakage, and tasks needing cross-frame integration (scrolling text, trajectory, scrambled text, multi-hop needle-in-haystack).
Key results
- Gemini-2.5 Pro achieves the best overall accuracy of 73.7% on MME-VideoOCR, while LLaVA-OneVision 7B is lowest at 46.0%.
- Most models score below 25% on Cross-Frame Text Understanding and 0% on Trajectory Recognition for the top-5 models.
- Increasing input resolution and frame count consistently improves performance across models; token-compression models (VideoChat-Flash, Slow-fast MLLM) lag.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model parameter scale | Qwen2.5-VL · MME-VideoOCR (overall) | accuracy: 59.1% -> 61.0% -> 69.0% | Mixed |
| Increase model parameter scale | InternVL3 · MME-VideoOCR (overall) | accuracy: 59.8% -> 66.1% -> 67.2% | Helped |
| Increase model parameter scale | Oryx-1.5 · MME-VideoOCR (overall) | accuracy: 49.6% -> 55.2% | Helped |
| Use higher input resolution | Qwen2.5-VL, VideoLLaMA 3, and Oryx-1.5 · MME-VideoOCR | accuracy | Helped |
| Increase the number of input video frames | Qwen2.5-VL, InternVL3, LLaVA-Video, and Oryx-1.5 · MME-VideoOCR | accuracy | Mixed |
| Use token compression for video processing | VideoChat-Flash and Slow-fast MLLM · MME-VideoOCR (overall) | accuracy | Hurt |
| Integrate information across multiple frames | Current MLLMs · Cross-Frame Text Understanding | accuracy | Hurt |
| Answer using information present in a few frames rather than scattered across multiple frames | Current MLLMs · Text-Based Video Understanding | accuracy | Mixed |
| Present questions and options without meaningful visual input | Qwen2.5-VL · MME-VideoOCR debiasing test | accuracy: 0% | No clear effect |
| Present questions and options with a black image instead of meaningful visual input | Qwen2.5-VL · MME-VideoOCR debiasing test | accuracy: 0% | No clear effect |
Surprising findings
- Top-5 models score 0% on Trajectory Recognition, and most models strongly favor language priors, correcting misspelled on-screen text to plausible words.
- Strong performance on general video multiple-choice benchmarks does not transfer; token-compression models perform poorly on MME-VideoOCR.
Limitations
- Only 2,000 QA pairs due to annotation and evaluation cost, so some subcategories have sparse samples and score fluctuations.
- Benchmark may become saturated as cutting-edge MLLMs already perform well on easy and medium questions.
- Frame counts and settings vary across models (e.g., GPT-4o limited to 50 frames, VITA-1.5 to 16), complicating comparison.
Experimental setup
Datasets: BOVText M4-ViteVQA NewsVideoQA LSVTD RoadText-1K RoadTextVQA EgoTextVQA NIAH-Video DSText
Benchmarks: MME-VideoOCR
Metrics: accuracy containment match accuracy GPT-assisted scoring accuracy multiple-choice accuracy
Baselines: GPT-4o Gemini-1.5 Pro Gemini-2.5 Pro Qwen2.5-VL InternVL3 LLaVA-OneVision LLaVA-Video VideoLLaMA 3 VideoChat-Flash Oryx-1.5 VITA-1.5 Kimi-VL
Builds on: VideoChat-Flash PaddleOCR
Closest papers by meaning
- OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
- PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks (CVPR 2026 · similarity 0.88)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.88)
- UPOCR: Towards Unified Pixel-Level OCR Interface (ICML 2024 · similarity 0.87)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.