Research map / Large language models / Language, safety and interpretability

Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Task vectors and steering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies activation engineering for hallucination mitigation in VideoLLMs and proposes a temporal-aware variant that routes each input to temporal-invariant or temporal-variant offset vectors chosen by a lightweight classifier.

Problem

Hallucination in video LLMs is under-addressed, and it is unclear whether activation engineering, which works for LLMs and image LLMs, transfers to VideoLLMs and which factors govern its effectiveness.

Approach

Hallucination sensitivity of attention heads/layers in VideoLLMs tracks the temporal variation of a task rather than its task type, so the framework builds separate offset-vector sets from temporal-invariant and temporal-variant datasets and uses a trained classifier to select which set to inject per input, without LLM fine-tuning.

Key results

• On VidHalluc, the method reaches 72.49 overall for VideoLLaMA2 versus 68.46 baseline, and 68.57 for Qwen2.5-VL versus 64.22 baseline.
• On EventHallusion, the method reaches 70.91 overall for Video-LLaVA versus 46.70 baseline, exceeding TCD (58.68) and DINO-HEAL (53.79).
• The temporal variation classifier assigns more than 97% of BQA/MCQ samples to temporal-invariant and more than 94% of STH/TSH samples to temporal-variant.

Measured effects reported in the paper

Surprising findings

• Attention heads within the same transformer layer show nearly identical hallucination-detection accuracy, and attention head engineering and layer engineering perform similarly, suggesting they may be intrinsically equivalent.
• Naively mixing datasets from tasks with different temporal variation characteristics yields only moderate gains across all tasks, compared with task-specific datasets.

Limitations

• Taxonomy is coarse, with only two temporal-variation categories, which oversimplifies heterogeneity within and across tasks.
• The method still adds some inference overhead despite pre-computation and overlapping the classifier with the forward pass.
• Evaluated on only two hallucination benchmarks and three VideoLLMs.

Experimental setup

Datasets: ShareGPT4Video VidHalluc EventHallusion

Benchmarks: VidHalluc EventHallusion

Metrics: accuracy overall score

Baselines: TCD DINO-HEAL Baseline (direct VideoLLM evaluation)

Builds on: ITI ICT VTI TCD

Closest papers by meaning

• Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models (ICML 2026 · similarity 0.94)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding (CVPR 2025 · similarity 0.92)
• MHBench: Demystifying Motion Hallucination in VideoLLMs (AAAI 2025 · similarity 0.92)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)