Research map / Video / Video-language understanding
MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MASH-VLM, a Video-LLM with disentangled spatial-temporal attention (DST-attention) and Harmonic-RoPE to reduce action-scene hallucination, and introduces the UNSCENE benchmark for evaluating it.
Problem
Video-LLMs hallucinate actions from scene context or scenes from observed actions (action-scene hallucination), because spatial and temporal tokens get entangled inside the LLM and standard RoPE biases text tokens toward whichever token type is closer in sequence order.
Approach
Within the LLM, a disentangling mask blocks direct attention between spatial and temporal tokens (causal among temporal, bidirectional among spatial, text attends to both), and Harmonic-RoPE adds a second set of balanced position IDs in alternating rotary dimensions so spatial and temporal tokens are equidistant from text tokens.
Key results
- On the UNSCENE binary task, MASH-VLM reaches 57.85% average, about 16.6 points above VideoChat2 (41.27%) and ST-LLM (35.15%).
- MASH-VLM scores 57.6% on MVBench, the best reported in the paper, versus 54.9% for ST-LLM.
- On VCG-Bench, MASH-VLM averages 4.46 versus 3.15 for ST-LLM, the strongest prior baseline in the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MASH-VLM instead of VideoChat2 | Video-LLM · UNSCENE Binary | average score: 41.27 -> 57.85 | Helped |
| Use MASH-VLM instead of ST-LLM | Video-LLM · UNSCENE Open-ended | average score: 2.9 -> 3.3 | Helped |
| Use MASH-VLM instead of ST-LLM | Video-LLM · MVBench | average score: 54.9 -> 57.6 | Helped |
| Use MASH-VLM instead of ST-LLM | Video-LLM · VCG-Bench | average score: 3.15 -> 4.46 | Helped |
| Use causal temporal and bidirectional spatial attention instead of causal attention for both token types (ablation) | DST-attention · UNSCENE Binary | score: 41.81 -> 41.89 | Helped |
| Use causal temporal and bidirectional spatial attention instead of bidirectional attention for both token types (ablation) | DST-attention · UNSCENE Binary | score: 41.18 -> 41.89 | Helped |
| Use causal temporal and bidirectional spatial attention instead of bidirectional temporal and causal spatial attention (ablation) | DST-attention · UNSCENE Binary | score: 41.14 -> 41.89 | Helped |
| Use causal temporal and bidirectional spatial attention instead of causal attention for both token types (ablation) | DST-attention · MVBench | score: 53.41 -> 55.29 | Helped |
| Use causal temporal and bidirectional spatial attention instead of bidirectional attention for both token types (ablation) | DST-attention · MVBench | score: 54.31 -> 55.29 | Helped |
| Use causal temporal and bidirectional spatial attention instead of bidirectional temporal and causal spatial attention (ablation) | DST-attention · MVBench | score: 53.11 -> 55.29 | Helped |
| Disable direct attention in both spatial-temporal directions instead of removing the disentangling mask (ablation) | DST-attention · UNSCENE Binary | score: 42.10 -> 45.36 | Helped |
| Disable direct attention in both spatial-temporal directions instead of disabling spatial-to-temporal attention only (ablation) | DST-attention · UNSCENE Binary | score: 41.89 -> 45.36 | Helped |
Surprising findings
- Harmonic-RoPE with balanced IDs alone gives most of the UNSCENE gain (50.80 vs 45.36 for distinct only), and combining both reaches 57.85.
Limitations
- UNSCENE is small (1,320 videos) and relies on manually collected YouTube clips and GPT-4-turbo-generated distractor labels.
- Open-ended evaluation uses GPT-3.5 as judge, which may be noisy.
- Ablations are reported on UNSCENE binary and MVBench only, with the Harmonic-RoPE and DST-attention contributions not fully separated on other benchmarks.
Experimental setup
Datasets: VideoChatGPT-100k VideoChat-11k Webvid NExT-QA CLEVRER Kinetics-710 Something-Something-2 MVBench VCG-Bench MSRVTT-QA MSVD-QA TGIF-QA
Benchmarks: UNSCENE MVBench VCG-Bench MSRVTT-QA MSVD-QA TGIF-QA ActivityNet-QA
Metrics: accuracy (dual-validation binary) GPT score (0-5) multiple-choice accuracy
Baselines: Video-ChatGPT VideoChat Chat-UniVi VideoChat2 ST-LLM Video-LLaMA GPT-4V LLaMA-VID Video-LLaVA LongVLM
Builds on: Video-ChatGPT RoPE CLIP Vicuna ST-LLM
Closest papers by meaning
- MHBench: Demystifying Motion Hallucination in VideoLLMs (AAAI 2025 · similarity 0.92)
- Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering (NeurIPS 2025 · similarity 0.91)
- VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding (CVPR 2025 · similarity 0.91)
- Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models (ICML 2026 · similarity 0.91)
- Mitigating Object Hallucination via Concentric Causal Attention (NeurIPS 2024 · similarity 0.90)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.