Research map / Video / Video-language understanding

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VideoLLM temporal-order sensitivity peaks at intermediate layers then fades, and proposes Temporal Activation Injection (TAI), a training-free method that reinjects the reversal-derived temporal divergence vector into later layers at the last token to improve temporal reasoning.

Problem

VideoLLMs often fail temporal reasoning: reversing frame order frequently leaves predictions unchanged, and the cause of this failure within the layer stack is unclear.

Approach

Define a temporal divergence vector as the last-token hidden-state difference between forward and time-reversed videos; its normalized norm peaks at mid-to-late layers and decays toward output. TAI extracts this vector at the peak and adds it to subsequent layers with profile-derived weights, requiring no training.

Key results

• TAI improves TempCompass average for Qwen2.5-VL-7B from 73.4 to 75.6 and for Qwen3-VL-8B from 77.1 to 78.0 (TempCompass AVG).
• On TVBench, TAI raises InternVL2.5-8B from 57.2 to 58.3 and Qwen3-VL-8B from 49.9 to 52.6.
• Anti-TAI (negated beta) drops Attribute Change accuracy by 36.3 points on Qwen2.5-VL-7B, while Action and Speed stay within about 1 point.

Surprising findings

• Reversing the injection sign (Anti-TAI) selectively degrades reversal-sensitive categories, supporting that the extracted vector encodes a temporal signal.
• On AoTBench, TAI applies near-zero injection on reversal-invariant subtasks because the input-level divergence magnitude is small.

Limitations

• Only addresses temporal information arising from frame order; other video reasoning factors are out of scope.
• Gains on MVBench temporal-relevant tasks are smaller than on TempCompass.
• Beta is a model-specific hyperparameter that must be set per model.

Experimental setup

Datasets: TempCompass TVBench AoTBench MVBench

Benchmarks: TempCompass TVBench AoTBench MVBench

Metrics: accuracy McNemar test

Baselines: TCD DINO-HEAL ArrowRL

Builds on: Representation Engineering contrastive decoding attention knockout

Closest papers by meaning

• EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.90)
• Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.89)
• On the Consistency of Video Large Language Models in Temporal Comprehension (CVPR 2025 · similarity 0.89)
• When Thinking Drifts: Evidential Grounding for Robust Video Reasoning (NeurIPS 2025 · similarity 0.89)
• ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
• Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.88)