Research map / Vision-language and multimodal / Multimodal LLMs
Mitigating Object Hallucination via Concentric Causal Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Concentric Causal Attention (CCA), which reorders visual tokens from periphery to center with matching concentric causal masks, to reduce RoPE long-term decay and mitigate object hallucination in LLaVA-style LVLMs.
Problem
Object hallucination in large vision-language models, which the authors link to long-term decay of rotary position encoding (RoPE) that weakens attention between distant visual tokens and instruction tokens.
Approach
Reorder visual tokens into concentric squares (peripheral to center) and rectify the causal mask to follow 2-D spatial locality, which shortens the relative distance between visual and instruction tokens so RoPE decay affects them less.
Key results
- On POPE, CCA-LLaVA improves accuracy over the LLaVA baseline by 5.48%, 7.86% and 6.70% across the three negative-sampling setups, and surpasses VCD by 1.02%, 4.51% and 2.65% on the three datasets.
- On CHAIR with greedy decoding and 512 max tokens, CCA-LLaVA reduces C_S from 46.2 to 43.0 while keeping recall at 80.4 vs 80.3 for the baseline.
- On MME hallucination subset, CCA-LLaVA reaches 641.66 total vs 565.33 for the baseline and 604.66 for VCD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · MSCOCO POPE average | accuracy: 81.38 -> 86.86 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · MSCOCO POPE average | F1: 79.65 -> 85.54 | Helped |
| Use Concentric Causal Attention instead of VCD | LLaVA · MSCOCO POPE average | accuracy: 84.66 -> 86.86 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · A-OKVQA POPE average | accuracy: 79.13 -> 86.99 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · GQA POPE average | accuracy: 78.99 -> 85.69 | Helped |
| Use Concentric Causal Attention instead of VCD | LLaVA · GQA POPE average | accuracy: 81.16 -> 85.69 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · CHAIR, greedy, 512 tokens | CHAIR sentence-level: 46.2 -> 43.0 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · CHAIR, greedy, 512 tokens | CHAIR instance-level: 12.9 -> 11.5 | Hurt |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · CHAIR, greedy, 64 tokens | CHAIR sentence-level: 21.0 -> 18.2 | Helped |
| Add Concentric Causal Attention to beam search with OPERA | LLaVA + OPERA · CHAIR, beam (5), 512 tokens | CHAIR sentence-level: 46.8 -> 45.0 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · MME hallucination subset | Total: 565.33 -> 641.66 | Helped |
| Replace raster-scan visual token alignment with Concentric Causal Attention | LLaVA · LLaVA-Bench | Overall: 58.9 -> 64.3 | Helped |
Surprising findings
- Reversing the raster-scan order of visual tokens alone degrades object hallucination performance, showing sensitivity to visual token order under RoPE.
Limitations
- Evaluation limited to image-text inputs; other modalities such as audio or video are not studied.
- No error bars are reported because LVLM training is computationally expensive.
- Pilot experiments use LoRA for 20K steps due to resource limits, not the full schedule.
Experimental setup
Datasets: CC-558K LLaVA 665K instruction data COCO VAL 2014 POPE GQA A-OKVQA LLaVA-Bench SEED-Bench ScienceQA VizWiz MMBench MMStar
Benchmarks: POPE CHAIR MME LLaVA-Bench SEED-Bench ScienceQA GQA VizWiz MMBench MMStar TextVQA
Metrics: accuracy F1 score CHAIR_S CHAIR_I recall caption length MME score GPT-4 rating
Baselines: LLaVA baseline VCD LLaVA-RLHF OPERA
Builds on: LLaVA RoPE VCD OPERA
Closest papers by meaning
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
- One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.93)
- Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.92)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.92)
- Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.