Research map / Video / Video-language understanding
CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CG-Bench, a long-video QA benchmark (1,219 videos, 12,129 QA pairs with clue intervals) with clue-grounded white-box, black-box, and clue-aided open-ended evaluations, and benchmarks closed and open MLLMs.
Problem
Existing long-video understanding benchmarks rely on multiple-choice questions, which MLLMs can often answer by elimination without genuinely grounding answers in video clues, so evaluation credibility is limited.
Approach
Annotate each question with human-labeled clue intervals and evaluate models on whether they localize those clues (white-box tIoU/acc@IoU), whether long-video and clue-only MCQ accuracy are consistent (black-box CRR), and use a clue-aided heuristic GPT-4o judge for open-ended QA.
Key results
- GPT-4o-0806 with 128 frames reaches 45.2 long-acc. on CG-Bench MCQ but only 4.38 acc.@IoU under the clue-grounded white-box metric.
- Qwen2-VL-72B achieves 41.3 long-acc. and 56.2 clue-acc., the strongest open-source result, comparable to GPT-4o on long-acc.
- Human evaluators with 128 sparse frames reach 59.9% MCQ accuracy versus 90.3% with full video, indicating undersampling severely limits performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add frame timestamps to frames-only prompts (ablation) | GPT-4o on CG-Bench · CG-Bench, subset with subtitles | Acc@IoU: 10.7 -> 20.4 | Helped |
| Add subtitles to frames-only prompts (ablation) | GPT-4o on CG-Bench · CG-Bench, subset with subtitles | long-acc.: 51.8 -> 53.4 | Helped |
| Add subtitle timestamps to frames and subtitles (ablation) | GPT-4o on CG-Bench · CG-Bench, subset with subtitles | long-acc.: 53.4 -> 54.1 | Helped |
| Add frame timestamps to frames and subtitles (ablation) | GPT-4o on CG-Bench · CG-Bench, subset with subtitles | Acc@IoU: 10.7 -> 22.3 | Helped |
| Add subtitles and frame/subtitle timestamps to frames (ablation) | GPT-4o on CG-Bench · CG-Bench, subset with subtitles | mIoU: 3.39 -> 9.68 | Helped |
| Add audio to frames, subtitles, and timestamps (ablation) | Gemini-1.5-Pro on CG-Bench · CG-Bench, audio modality experiment | Acc@IoU: 20.7 -> 19.8 | Hurt |
| Add audio to frames, subtitles, and timestamps (ablation) | Gemini-1.5-Pro on CG-Bench · CG-Bench, audio modality experiment | OE-acc.: 23.2 -> 23.5 | Helped |
| Increase the number of sampled frames (ablation) | GPT-4o, Gemini-1.5-Pro, and Qwen2-VL · CG-Bench | long-acc. | Helped |
| Use higher-resolution frames instead of low-resolution frames (ablation) | GPT-4o-08-06 · CG-Bench, 50 uniformly sampled frames | long-acc. | Helped |
| Use FFmpeg keyframe extraction with low resolution (ablation) | GPT-4o-08-06 · CG-Bench, 50 uniformly sampled frames | long-acc. | No clear effect |
| Evaluate open-ended answers using the proposed heuristic with clue-based visual requests | GPT-4o evaluator · CG-Bench open-ended QA | evaluation bias | Helped |
Surprising findings
- Adding subtitles and timestamps helps interval prediction, but audio input gives no significant gain and slightly degrades results for Gemini-1.5 Pro.
- Keyframe extraction via FFmpeg has no significant impact on long-video MCQ accuracy, while higher resolution offers some improvement.
Limitations
- Human MCQ evaluation was done on a limited subset (30 videos, 296 questions) under constrained 128-frame sampling.
- Open-ended evaluation depends on GPT-4o as judge, which may still introduce hallucination errors.
- Many models were limited by frame count (e.g., 128 frames) due to hardware constraints.
Experimental setup
Benchmarks: CG-Bench
Metrics: clue-acc. long-acc. mIoU rec.@IoU acc.@IoU CRR open-ended acc.
Baselines: GPT-4o Gemini-1.5 Pro Gemini-1.5 Flash Claude3.5-Sonnet Qwen2-VL-72B InternVL2 LLaVA-OneVision MiniCPM-V2.6 LongVA
Builds on: NExT-GQA Video-MME MLVU MM-Vet
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.91)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.