Research map / Vision-language and multimodal / Multimodal LLMs
MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoLE, a training-free layer-wise decoding method that sums logits from three expert layers (Final, Second Opinion, Prompt Retention) of an LVLM to reduce object hallucinations.
Problem
Object hallucinations in large vision-language models during generation, including hallucinations from late-layer knowledge injection and forgetting of the prompt in long outputs.
Approach
Instead of contrasting against an amateur model as in contrastive decoding, MoLE treats different layers of the same LVLM as experts selected by heuristic gating (JSD divergence on top-k tokens for the second opinion; attention mass on prompt tokens for prompt retention) and sums their logits in a single forward pass.
Key results
- On POPE Random with MiniGPT-4, MoLE reaches 0.818 accuracy vs 0.750 for Beam Search, an 8.7% gain.
- On CHAIR, MoLE gives CHAIRS 18.0 and CHAIRI 6.00 on LLaVA-1.5 vs 22.20 / 7.40 for Beam Search.
- MiniGPT-4 CHAIRI is reduced about 21% relative to DoLA according to the text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoLE decoding | MiniGPT-4 · POPE Random | accuracy | Helped |
| Use MoLE decoding | LLaVA-1.5 · POPE Random | accuracy | Helped |
| Use MoLE decoding | Shikra · POPE Adversarial | accuracy | Helped |
| Use MoLE instead of prior decoding methods | MiniGPT-4 · CHAIR | CHAIRI: 8.20 -> 7.10 | Helped |
| Add the Second Opinion expert (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 22.2 -> 22.2 | No clear effect |
| Add the Second Opinion expert (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRI: 7.4 -> 7.1 | Helped |
| Add SO gating (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 22.2 -> 20.4 | Helped |
| Add the Prompt Retention expert (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 22.2 -> 19.4 | Helped |
| Add PR gating (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 19.4 -> 18.0 | Helped |
| Use SO gating instead of averaging candidate-layer logits (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 22.2 -> 17.2 | Helped |
| Use SO gating instead of random layer selection (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRS: 23.2 -> 17.2 | Helped |
| Use SO gating instead of selecting the penultimate layer (ablation) | MoLE on LLaVA-1.5 · CHAIR | CHAIRI: 8.2 -> 5.63 | Helped |
Surprising findings
- Averaging logits across candidate layers or using a fixed layer for the second opinion expert performs worse than the dynamic gating, indicating layer choice matters.
Limitations
- Evaluation limited to POPE and CHAIR on 600 and 500 sampled items, with a small image subset.
- Gating hyperparameters (k, alpha, lambda) were tuned on the same evaluation setup.
- Experiments only on 7B-scale Vicuna-based LVLMs.
Experimental setup
Datasets: MSCOCO
Benchmarks: POPE CHAIR
Metrics: accuracy precision F1 score CHAIRS CHAIRI
Baselines: Greedy Search Beam Search OPERA VCD DoLA
Builds on: DoLA Mixture of Experts contrastive decoding VCD OPERA
Closest papers by meaning
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.94)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.93)
- Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.