Research map / Vision-language and multimodal / Multimodal LLMs
Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemVR, a training-free decoding method that re-injects visual tokens into the MLLM's middle-layer FFN as key-value memory when next-token uncertainty is high, reducing hallucinations without extra inference passes.
Problem
MLLMs hallucinate, with the text decoder becoming increasingly insensitive to visual tokens in deeper layers ("amnesia" about visual information); existing fixes like contrastive decoding or attention intervention add latency or hurt general performance.
Approach
Treat visual tokens as key-value memory in the FFN and re-inject them at a middle trigger layer, activated only when layer-wise entropy exceeds a threshold (look-twice), so no logit contrast or extra forward pass is needed.
Key results
- On POPE with LLaVA-1.5-7B, MemVR raises average accuracy on A-OKVQA by about +7.1 points over the base model (86.21 vs 79.13).
- On CHAIR, MemVR reduces CHAIR_I from 15.4 to 13.0 and CHAIR_S from 50.0 to 46.6 for LLaVA-1.5, while recall rises from 77.1 to 80.8.
- MemVR adds about 32 MME total points for LLaVA-1.5-7B (1896.72 vs 1864.68) with latency comparable to greedy decoding (68.32 vs 65.71 ms/token), unlike VCD (144.62 ms/token).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual retracing | LLaVA-1.5-7B · POPE COCO Average | Accuracy: 79.83 -> 86.93 | Helped |
| add visual retracing | LLaVA-1.5 · CHAIR | CHAIR_I: 15.4 -> 13.0 | Helped |
| add visual retracing | LLaVA-1.5 · HallusionBench | hardA: 36.7 -> 37.7 | Helped |
| add visual retracing | LLaVA-1.5 · LLaVA-Bench (in-the-wild) | Average: 64.8 -> 65.2 | Helped |
| add visual retracing | Qwen-VL-Chat · MME | Overall: +36.0 | Helped |
| add visual retracing | LLaVA-1.5 · HallusionBench | fACC: 17.9 -> 17.9 | No clear effect |
| add dynamic VR instead of fixed layer 7 (ablation) | MemVR on LLaVA-1.5-7B · MME | Total score: 1847.6 -> 1896.7 | Helped |
| use fixed layer 15 instead of dynamic VR (ablation) | MemVR on LLaVA-1.5-7B · MME | Total score: 1896.7 -> 1881.2 | Hurt |
| use fixed layer 23 instead of dynamic VR (ablation) | MemVR on LLaVA-1.5-7B · MME | Total score: 1896.7 -> 1858.1 | Hurt |
| derive injection ratio dynamically as α =2(u−γ) (ablation) | MemVR on LLaVA-1.5 · LLaVA-Bench | score: 65.17 -> 65.87 | Helped |
| derive injection ratio dynamically as α =2(u−γ) (ablation) | MemVR on LLaVA-1.5 · MME (Overall) | score: 1896.72 -> 1894.14 | Hurt |
| vary threshold γ from 0.6 to 0.95 (ablation) | MemVR on LLaVA-1.5-7B · MME | performance | Helped |
Surprising findings
- Re-injecting only image tokens gives the best gain in the look-twice study, while text or text+image re-injection helps less.
- Contrastive decoding methods such as ICD and VCD often degrade general benchmarks such as MME and LLaVA-Bench despite hallucination gains.
Limitations
- Hyperparameters (trigger threshold gamma, injection ratio alpha, candidate layer bucket) need tuning, with the layer choice sensitive to data distribution.
- Failure cases where the base model is correct but MemVR's re-injection disturbs reasoning, e.g., in MME celebrity, scene, and landmark tasks.
- Authors state code, datasets, and tutorials will be released; the release is promised rather than yet confirmed.
Experimental setup
Datasets: MSCOCO val2014 A-OKVQA GQA
Benchmarks: POPE CHAIR HallusionBench MME MMBench MM-Vet VizWiz-VQA LLaVA-Bench
Metrics: accuracy F1-score CHAIR_S CHAIR_I latency throughput memory cost
Baselines: OPERA VCD ICD DoLa
Builds on: DoLa Visual Contrastive Decoding OPERA early exit
Closest papers by meaning
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.93)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)
- One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
- Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.