Research map / Video / Video-language understanding

Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video LLMs

NeurIPS 2026 · Video › Video-language understanding · approach: Event and anomaly reasoning in video · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies directional motion blindness in Video-LLMs, traces it to a direction binding gap and an OOD magnitude deficit, and proposes DeltaDirect, a training-only projector-level motion-vector prediction loss trained with the MoDirect dataset family.

Problem

Video-LLMs near chance on signed image-plane motion direction (left/right/up/down), even though direction is linearly decodable from internal representations; instruction tuning fixes source-domain binding but not out-of-domain generalization.

Approach

Diagnose the failure via linear probes, logit lens and concept-vector analysis to locate a binding gap and magnitude deficit, then add a training-only loss that predicts normalized 2-D motion vectors from adjacent-frame projector feature deltas, leaving inference unchanged.

Key results

• On MoDirect-SynBench, LLaVA-Video-7B with MoDirect-Inst tuning and DeltaDirect reaches 85.9% average accuracy, up from 25.9% vanilla.
• On MoDirect-RealBench, DeltaDirect improves average accuracy from 58.6% (Inst tuning only) to 64.5%, and the paper reports a 21.4-point gain over vanilla without real-world tuning data.
• DeltaDirect keeps general video performance on par for LLaVA-Video-7B (standard average 69.4 to 69.8; fine-grained 47.3 to 48.5).

Surprising findings

• Most Video-LLMs perform near chance on signed motion direction, and above-chance results are largely due to prediction biases.
• Supervising the final readout token with the motion loss performs poorly (SynBench 27.7), even though the binding gap is observed there.

Limitations

• Synthetic training data is restricted to single-object linear-like motion in four cardinal directions.
• Real-world evaluation uses small curated subsets (SSv2, KTH, TOMATO) reformulated as binary or MCQ tasks.
• Gains on general video benchmarks are small and mixed (e.g., NExT-QA drops slightly for LLaVA-Video-7B).

Experimental setup

Datasets: MoDirect-Inst MoDirect-SynBench MoDirect-RealBench VideoChat2-IT LLaVA-Video-178K Something-Something-V2 KTH TOMATO

Benchmarks: MoDirect-SynBench MoDirect-RealBench MVBench NExT-QA Perception Test EgoSchema TGIF-QA TempCompass Vinoground FAVOR-Bench MotionBench

Metrics: top-1 accuracy cosine similarity logit lens accuracy

Baselines: GPT-4o Gemini 2.5 Flash Qwen2.5-VL-7B Qwen2.5-VL-72B Qwen3-VL-4B LLaVA-OneVision-7B LLaVA-Video-72B VideoLLaMA3-7B Motion-o-7B MotionSight-7B ReMoRa-7B MotionLLM-7B

Builds on: LLaVA-Video difference-in-means concept vectors logit lens LoRA

Closest papers by meaning

• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
• HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.88)
• PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? (NeurIPS 2026 · similarity 0.87)
• Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.86)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)
• Towards Understanding Camera Motions in Any Video (NeurIPS 2025 · similarity 0.86)