Research map / Vision-language and multimodal / Visual question answering

Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models

NeurIPS 2026 · Vision-language and multimodal › Visual question answering · approach: Multimodal question answering datasets

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces three vision-centric VQA benchmarks (temporal frame retrieval, video future prediction, causal memory distortion) where answer options are video frames rather than text, and shows frontier video models perform poorly on them.

Problem

Video QA benchmarks evaluate large video models almost exclusively with text multiple-choice options, which may hide weak visual grounding and allow language shortcuts.

Approach

Replace text answer options with visual options (frame sequences from the video or simulator) and add a cognitive-science-inspired causal memory distortion test, so the evaluation modality itself is visual.

Key results

• Gemini-2.5-Pro reaches 30.00% on temporal frame retrieval with visual options versus 73.87% with text options, near the 25% random baseline.
• Gemini-2.5-Pro scores 69.62% on video future prediction, the best result, while most open models stay near 25%; Kimi-K2.6 is best on temporal frame retrieval at 33.45%.
• On causal memory distortion, Qwen2.5-72B scores 60% accuracy vs Gemini-2.5-Pro's 58%, with Gemini's specificity about 30 points lower, indicating more false 'yes' hallucinations.

Surprising findings

• Distilling Shapestacks visual options into GPT-5.4 text captions drops Gemini accuracy from 69.62% to 45.12%, showing textual conversion loses visual detail.
• Human accuracy on temporal frame retrieval is only 44%, below the 80% human scores on the other two tasks.

Limitations

• Small dataset sizes: causal memory distortion uses only 20 source videos and 60 clips.
• Analysis is entirely black-box with no examination of model internals or vision encoders.
• Human-model comparison is not controlled because humans have memory that models lack.

Experimental setup

Datasets: Charades Shapestacks VideoCon

Benchmarks: Temporal Frame Retrieval (Charades-based) Video Future Prediction (Shapestacks-based) Causal Memory Distortion (Charades-based)

Metrics: accuracy sensitivity specificity d-prime response bias perplexity

Baselines: Random Baseline Human Gemini-2.5-Pro GLM-4.6V Qwen2.5-72B Qwen2.5-32B Qwen3-32B LLaVA-OV-72B InternVL3-72B mPLUG-Owl3 Kimi-K2.6 GLM-5V-Turbo

Builds on: TemporalBench TempCompass MVBench LVBench

Closest papers by meaning

• Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.91)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
• VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.91)
• Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.91)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.90)
• When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)