Research map / Video / Video-language understanding
VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VISTA-LLAMA, a frozen-LLaVA video-LM that removes RoPE from visual-text attention (EDVT attention) and adds a sequential Q-Former projector, training only the projector to reduce video hallucination.
Problem
Video-language models generate text unrelated to the video as text length grows and visual influence diminishes (hallucination), and long videos strain LLM context length.
Approach
Equal-distance-to-visual-tokens (EDVT) attention applies rotary position embedding only among text tokens and omits it for visual-text attention weights, so visual tokens keep a constant influence regardless of text length; a sequential visual projector conditions each frame's tokens on the previous frame's projected tokens to model temporal structure with fewer tokens.
Key results
- Achieves 60.7% accuracy on zero-shot NExT-QA, the reported state of the art over Video-ChatGPT (54.6%).
- Achieves 60.5% accuracy on zero-shot MSRVTT-QA, versus 49.7% for MovieChat, the second-best model.
- Adding EDVT-Attention to Video-ChatGPT raises NExT-QA causal accuracy from 65.1% to 72.8% and average from 54.6% to 59.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EDVT-Attention (ablation) | Video-ChatGPT · NExT-QA | accuracy: 54.6 -> 59.3 (+4.7) | Helped |
| Add EDVT-Attention (ablation) | VISTA-LLAMA (ViT-L-14) · NExT-QA | accuracy: 53.6 -> 56.5 (+2.9) | Helped |
| Add EDVT-Attention (ablation) | VISTA-LLAMA (EVA-CLIP-g) · NExT-QA | accuracy: 54.1 -> 59.7 (+5.6) | Helped |
| Replace BERT initialization with BLIP-2 initialization for Q-Former (ablation) | VISTA-LLAMA · NExT-QA | accuracy: 51.8 -> 54.1 | Helped |
| Add the sequential design to the BLIP-2-initialized Q-Former (ablation) | VISTA-LLAMA · NExT-QA | accuracy: 54.1 -> 55.4 | Helped |
| Replace the frame linear projector with a BLIP-2-initialized Q-Former (ablation) | VISTA-LLAMA · NExT-QA | accuracy: 54.6 -> 54.1 | Hurt |
| Add EDVT-Attention (ablation) | VISTA-LLAMA · NExT-QA temporal questions | accuracy: 34.3 -> 40.7 (+6.4) | Helped |
| Add EDVT-Attention (ablation) | VISTA-LLAMA · NExT-QA causal questions | accuracy: 65.8 -> 72.3 (+6.5) | Helped |
Surprising findings
- Temporal-question accuracy on NExT-QA declines as the number of training visual tokens increases, even though overall accuracy improves.
Limitations
- Built on a pre-trained image-text architecture; gains could be larger if applied during video pre-training (authors' own suggestion).
- Evaluation relies on GPT-3.5 judge with relative scores, and the sequential encoder's performance on longer videos is not yet assessed.
Experimental setup
Datasets: Video-ChatGPT 100K video instruction pairs NExT-QA MSVD-QA MSRVTT-QA ActivityNet-QA ActivityNet-200
Benchmarks: NExT-QA MSVD-QA MSRVTT-QA ActivityNet-QA ActivityNet-200 video-based text generation
Metrics: accuracy score (0-5) correctness consistency detail orientation contextual understanding temporal understanding
Baselines: FrozenBiLM Video Chat LLaMA Adapter Video LLaMA MovieChat Video-ChatGPT
Builds on: LLaVA BLIP-2 Video-ChatGPT RoPE Vicuna
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations (CVPR 2025 · similarity 0.89)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.89)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.89)
- The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.