Research map / Video / Video-language understanding
Look, Remember and Reason: Grounded Reasoning in Videos with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an OPT-based multimodal LM with a two-stream video encoder end-to-end on low-level surrogate tasks (detection, re-identification, tracking) so it can reason over videos grounded in low-level visual details.
Problem
Multimodal language models struggle with causal and compositional spatiotemporal video reasoning that requires grounding in fine-grained low-level details such as object motion and interactions.
Approach
Train the LM end-to-end with randomly inserted low-level surrogate tasks (object recognition, re-identification, tracking) emitted as text, and feed video through a two-stream (spatial + temporal) encoder whose features are injected into the LM via top-down cross-attention, so reasoning is grounded in low-level visual skills.
Key results
- On ACRE, LRR reaches 98.2% (compositional) and 99.2% (systematic) vs. ALOE's 91.7% and 93.9%.
- On CATER moving camera, LRR achieves 80.4% top-1 vs. ALOE's 59.7%; static camera 84.1% vs. TFC V3D's 79.7%.
- On STAR validation, LRR reaches 70.5% overall accuracy vs. SeViLA's 64.9%; on Something-Else compositional, 62.0% top-1 vs. STIN+OIE+NL's 56.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add surrogate tasks (ablation) | LRR · ACRE compositional | accuracy: 38.1 -> 98.2 | Helped |
| add surrogate tasks (ablation) | LRR · ACRE systematic | accuracy: 36.5 -> 99.2 | Helped |
| use a two-stream encoder instead of a single-stream ViT (ablation) | LRR · ACRE compositional | accuracy: 92.2 -> 98.2 | Helped |
| initialize from a pretrained LM instead of training from scratch (ablation) | LRR · ACRE compositional | accuracy: 87.7 -> 98.2 | Helped |
| add surrogate tracking tasks (ablation) | LRR · Something-Else Compositional | Top-1 accuracy: 50.1 -> 62.0 | Helped |
| use a two-stream encoder instead of a single-stream ViT (ablation) | LRR · Something-Else Compositional | Top-1 accuracy: 53.6 -> 62.0 | Helped |
| add surrogate tracking tasks (ablation) | LRR · CATER moving camera | Top-1 accuracy: 62.7 -> 80.4 | Helped |
| use a two-stream encoder instead of a single-stream ViT (ablation) | LRR · CATER moving camera | Top-1 accuracy: 75.6 -> 80.4 | Helped |
| add surrogate tasks (ablation) | LRR · STAR validation | Overall accuracy: 48.2 -> 70.5 | Helped |
| prompt surrogate tasks after every frame during training and inference instead of random 30% prompting during training only (ablation) | LRR · ACRE compositional | accuracy: 98.2 -> 99.3 | Helped |
| prompt surrogate tasks after every frame during training and inference instead of random 30% prompting during training only (ablation) | LRR · ACRE | inference speed: 61 -> 1415 | Hurt |
| train jointly on CATER and Something-Else | LRR · Something-Else compositional | Top-1 accuracy: 56.2 -> 61.1 | Helped |
Surprising findings
- Removing surrogate tasks drops ACRE compositional accuracy from 98.2% to 38.1%, near chance-level behavior.
- Random 30% surrogate prompting during training matches the every-frame chain-of-thought-style variant (98.2 vs 99.3 on ACRE compositional) while inference is roughly 23x faster (61 ms vs 1415 ms).
Limitations
- Relies on small OPT backbones (125M-1.3B) and small training sets, compared to large pretrained video-language models.
- Surrogate tasks require ground-truth annotations (object tracks, boxes, scene graphs) or off-the-shelf vision models to generate them.
- Evaluated on specific synthetic or narrow benchmarks; generalist joint training gains are modest on some splits.
Experimental setup
Datasets: ACRE CATER Something-Else STAR Kinetics Moments in Time
Benchmarks: ACRE Something-Else CATER STAR
Metrics: accuracy (top-1) top-5 accuracy L1 grid distance overall accuracy
Baselines: NS-OPT IV-CL ALOE OpenFlamingo STIN + OIE + NL Video-ChatGPT R3D + NL LSTM OPNet Hopper TFC V3D Depthwise Internvideo BLIP-2
Builds on: OPT divided space-time attention (TimeSformer) ALOE Video-ChatGPT
Closest papers by meaning
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.90)
- Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding (CVPR 2026 · similarity 0.90)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.90)
- Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.89)
- OpenCoF: Learning to Reason Through Video Generation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.