Research map / Vision-language and multimodal / Multimodal LLMs
Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Octopus adaptively selects, per sample and per generation step, which contrastive decoding strategy (VCD, M3ID, AVISC, or none) to apply to a frozen LVLM, with a learned decision token trained via DPO to reduce hallucination.
Problem
Contrastive decoding methods for reducing hallucination in LVLMs apply one fixed perturbation to all samples and generation steps, although the causes of hallucination vary across samples and tokens.
Approach
A learnable "eye" token passed through a small transformer block plus an MLP chooses, at each generation step, one CD strategy (or none) as a "tentacle"; this policy is trained with DPO on preference workflows built from CHAIR scores while the LVLM stays frozen.
Key results
- On AMBER with LLaVA-1.5-7B, Octopus reaches CHAIR 4.8 versus 8.0 for the base model and 6.0 for the best single CD baseline (M3ID).
- On POPE MSCOCO (ALL) with LLaVA-1.5-7B, Octopus achieves 85.79 accuracy and 83.44 F1 versus 82.04/80.42 for the base model.
- On Object-HalBench with LLaVA-1.5-7B, Octopus obtains CHAIR_s 20.8 and CHAIR_i 6.6, improving over the base model's 25.0 and 9.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic Octopus decoding instead of the strongest listed single CD baseline | LLaVA-v1.5-7B · AMBER | CHAIR: 6.0 -> 4.8 | Helped |
| use dynamic Octopus decoding instead of AVISC | LLaVA-1.5-7B · AMBER discrimination | Accuracy: 70.70 -> 76.70 | Helped |
| use Octopus instead of ICD | LLaVA-1.5-7B · POPE_MSCOCO Random | Accuracy: 87.51 -> 87.51 | No clear effect |
| randomly select among three CD strategies at each generation step (ablation) | LLaVA-1.5-7B · AMBER | CHAIR: 8.0 -> 6.9 | Helped |
| add Octopus selection over VCD and M3ID (ablation) | LLaVA-1.5-7B · AMBER | CHAIR: 6.9 -> 5.5 | Helped |
| add Octopus selection over VCD and AVISC (ablation) | LLaVA-1.5-7B · AMBER | CHAIR: 6.9 -> 5.7 | Helped |
| add Octopus selection over M3ID and AVISC (ablation) | LLaVA-1.5-7B · AMBER | CHAIR: 6.9 -> 5.5 | Helped |
| add the third CD strategy to Octopus's candidate set (ablation) | Octopus on LLaVA-1.5-7B · AMBER | CHAIR: 5.5 -> 4.8 | Helped |
| use Cover rather than CHAIR as the preference criterion (ablation) | Octopus on LLaVA-1.5-7B · AMBER | Cover: 49.2 -> 50.1 | Helped |
| use the Average criterion (ablation) | Octopus on LLaVA-1.5-7B · AMBER | CHAIR: 4.8 -> 5.0 | Hurt |
| optimize with Monte Carlo sampling instead of the reported default (ablation) | Octopus on LLaVA-1.5-7B · AMBER | CHAIR: 4.8 -> 5.1 | Hurt |
| optimize with PPO instead of the reported default (ablation) | Octopus on LLaVA-1.5-7B · AMBER | CHAIR: 4.8 -> 5.8 | Hurt |
Surprising findings
- Each single CD strategy helps only a subset of samples, with only about 10% of cases effective across all three methods on the discriminative task.
- Random selection among CD strategies at each step already reduces hallucination to some extent in the ablation.
Limitations
- Decision module and CD candidate pool are limited to three existing CD strategies plus a null action.
- Preference data construction relies on CHAIR scores and random action sampling, which may be noisy.
- Evaluated mainly on LLaVA-1.5-7B and InstructBLIP with specific benchmarks.
Experimental setup
Datasets: MSCOCO AMBER Object-HalBench MMHalBench POPE
Benchmarks: AMBER Object-HalBench MMHalBench POPE
Metrics: CHAIR Cover HalRate Cog GPT-4 score Accuracy F1
Baselines: VCD M3ID AVISC LCD ICD OPERA HA-DPO EOS POVID HALVA HACL ConVis
Builds on: visual contrastive decoding M3ID AVISC direct preference optimization
Closest papers by meaning
- CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models (NeurIPS 2024 · similarity 0.93)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.92)
- Hallucination-aware Intermediate Representation Editing in Large Vision-Lanugage Models (ICLR 2026 · similarity 0.92)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.92)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.92)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.