Research map / Vision-language and multimodal / Multimodal LLMs
Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Nullu identifies a low-rank "HalluSpace" from SVD of hidden-feature differences between hallucinated and truthful prompts, then orthogonally projects MLP weights of LVLMs into its null space to reduce object hallucination with no inference overhead.
Problem
Object hallucination in large vision-language models, where generated text describes objects not present in the image; prior fixes require costly fine-tuning or add inference-time cost.
Approach
Extract a low-rank subspace from the difference of hidden features between hallucinated and truthful text prompts (paired on the same image), then edit model weights by projecting them onto the null space of that subspace, so the edited model needs no extra inference cost and suppresses LLM-prior-driven hallucinations.
Key results
- On LLaVA-1.5, Nullu reduces CHAIR_S from 20.40 (greedy) to 15.20 and CHAIR_I from 7.08 to 5.30 while keeping BLEU near 15.7.
- Nullu achieves best OPOPE F score on all three LVLMs (LLaVA-1.5 91.79, MiniGPT-4 92.07, mPLUG-Owl2 90.80) versus baselines.
- Nullu runs at the same throughput as greedy LLaVA-1.5 and about 7x faster than OPERA while adding no inference cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Nullu weight projection | LLaVA-1.5 · MSCOCO val, CHAIR | CHAIRS: 16.90±2.10 -> 15.20±0.60 | Helped |
| Apply Nullu weight projection | MiniGPT-4 · MSCOCO val, CHAIR | CHAIRS: 25.20±2.00 -> 21.40±1.00 | Helped |
| Apply Nullu weight projection | mPLUG-Owl2 · MSCOCO val, CHAIR | CHAIRS: 18.80±1.20 -> 15.60±1.20 | Helped |
| Apply Nullu weight projection | LLaVA-1.5 · MSCOCO val, OPOPE | Accuracy: 77.87±0.22 -> 79.52±0.04 | Helped |
| Apply Nullu weight projection | MiniGPT-4 · MSCOCO val, OPOPE | F score: 90.95±0.42 -> 92.07±0.65 | Helped |
| Apply Nullu weight projection | mPLUG-Owl2 · MSCOCO val, OPOPE | F score: 88.12±0.99 -> 90.80±0.52 | Helped |
| Apply Nullu weight projection | LLaVA-1.5 · LLaVA-Bench | Accuracy: 5.89 -> 6.53 | Helped |
| Edit layers 24-32 rather than 16-32 (ablation) | Nullu on LLaVA-1.5-7B · MSCOCO val, CHAIR | CHAIRS: 15.2 -> 16.2 | Hurt |
| Edit layers 28-32 rather than 16-32 (ablation) | Nullu on LLaVA-1.5-7B · MSCOCO val, CHAIR | CHAIRS: 15.2 -> 18.2 | Hurt |
| Edit layers 31-32 rather than 16-32 (ablation) | Nullu on LLaVA-1.5-7B · MSCOCO val, CHAIR | CHAIRS: 15.2 -> 16.4 | Hurt |
| Use rank 2 instead of rank 4 (ablation) | Nullu on LLaVA-1.5-7B · MSCOCO val, CHAIR | CHAIRS: 15.2 -> 20.2 | Hurt |
| Use rank 8 instead of rank 4 (ablation) | Nullu on LLaVA-1.5-7B · MSCOCO val, CHAIR | CHAIRS: 15.2 -> 17.8 | Hurt |
Surprising findings
- Nullu also improves MME OCR and code reasoning subtasks, which the authors attribute to a DPO-like effect.
- Lower-rank HalluSpace (k=4) works best; k=2 and k=16 both perform worse on CHAIR.
Limitations
- No theoretical argument for optimal layer set or rank k; chosen by hyperparameter sweep.
- Gains over HALC on CHAIR_I are moderate; results vary across models and settings.
Experimental setup
Datasets: LURE paired data MSCOCO val
Benchmarks: CHAIR POPE OPOPE MME LLaVA-Bench
Metrics: CHAIR_S CHAIR_I BLEU accuracy precision F score MME score GPT-4V aided accuracy detailedness
Baselines: Beam Search DoLa OPERA VCD Woodpecker LURE HALC
Builds on: LURE VCD DPO ITI OPERA
Closest papers by meaning
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.92)
- Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.92)
- HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
- Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.92)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.