Research map / Vision-language and multimodal / Multimodal LLMs
CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CODE, a training-free contrastive decoding method that contrasts next-token logits from the image against logits conditioned on the model's own detailed image description, with dynamic per-token weighting to reduce hallucination in LMMs.
Problem
Hallucination in large multi-modal models: generated responses are plausible but inconsistent with the visual content.
Approach
Use the model's self-generated comprehensive description as the contrastive counterpart to the image (instead of distorted images or noise), with a dynamic weight alpha_t = 1 - bounded divergence between image and description next-token distributions, and an adaptive plausibility threshold beta_t equal to the same divergence.
Key results
- On MMVP, CODE improves average accuracy over greedy decoding for LLaVA-1.5 13B from 30.67 to 34.00 and for InternVL 1.5 26B from 48.00 to 51.33.
- On MMHal-Bench, CODE raises the overall score for InternVL 26B from 3.15 to 3.52 and lowers the hallucination rate from 33.33 to 30.21.
- CODE is roughly 2-3x slower than VCD in decoding latency (e.g., 272.92 vs 177.99 ms/token for 7B-class models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use self-generated descriptions as contrastive references instead of standard decoding | LMMs with CODE · POPE adversarial | accuracy | Helped |
| use self-generated descriptions as contrastive references instead of standard decoding | LMMs with CODE · MMVP | average accuracy | Helped |
| add adaptive information constraint (dynamic β) (ablation) | CODE · MMVP | overall score: 28.67 -> 32.00; 32.67 -> 41.33; 45.33 -> 48.00 | Helped |
| add adaptive information constraint (dynamic β) (ablation) | CODE · LLaVA-Bench | overall score: 66.4 -> 67.9; 76.0 -> 81.6; 81.0 -> 83.5 | Helped |
| add dynamic restriction (dynamic α) (ablation) | CODE · MMVP | overall score: 28.67 -> 29.33; 32.67 -> 37.33; 45.33 -> 48.67 | Helped |
| add dynamic restriction (dynamic α) (ablation) | CODE · LLaVA-Bench | overall score: 66.4 -> 69.0; 76.0 -> 79.7; 81.0 -> 82.4 | Helped |
| add both dynamic restriction and adaptive information constraint (ablation) | CODE · MMVP | overall score: 28.67 -> 34.00; 32.67 -> 42.67; 45.33 -> 51.33 | Helped |
Surprising findings
- Description-only inputs (replacing the image with the self-generated description) perform sub-optimally on their own, yet serve as a useful contrastive reference.
Limitations
- Adds computational overhead (lower throughput and higher latency than vanilla decoding, and than VCD).
- Cannot fully eliminate hallucinations; failure cases biased toward language priors.
- Relies on the assumption that the self-generated description is not too deviated from the true answer.
Experimental setup
Datasets: MMVP POPE LLaVA-Bench (In-the-Wild) LLaVA-QA90 MMHal-Bench RealworldQA
Benchmarks: POPE MMVP LLaVA-QA90 MMHal-Bench LLaVA-Bench (In-the-Wild) RealworldQA
Metrics: accuracy precision F1 paired accuracy score ratio GPT-4 rating score hallucination rate throughput latency
Baselines: Greedy decoding Beam search Nucleus sampling OPERA VCD
Builds on: Contrastive Decoding VCD OPERA Bounded Divergence
Closest papers by meaning
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.94)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.94)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
- Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding (CVPR 2025 · similarity 0.93)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.93)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.