Research map / Detection and segmentation / Open-set and few-shot detection
GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLSim, a training-free object hallucination detector for LVLMs that fuses a global similarity (last instruction token vs. object token) with a local similarity (top-K patches selected by visual logit lens vs. object token) score.
Problem
Detecting object existence hallucinations in large vision-language models at the object level without external judges, ground truth, or training; prior self-evaluation scores use only global or only local signals.
Approach
Combine a global embedding similarity between the object token and the final prompt token with a local similarity between the object token and top-K image tokens selected via a visual logit lens, using a weighted sum (w=0.6).
Key results
- On MSCOCO with LLaVA-1.5-7B, GLSim reaches 83.7 AUROC versus 75.4 for Contextual Lens and 72.9 for Internal Confidence.
- On MSCOCO with Shikra, GLSim gains 12.7 AUROC points over SVAR (83.0 vs 70.7).
- Against POPE on MSCOCO with LLaVA-1.5-7B, GLSim gives 81.8 accuracy at 1.6 s per image versus 79.5 accuracy at 8.1 s for POPE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine global and local similarity rather than use either signal alone (ablation) | GLSim · MSCOCO | AUROC: 2.7% in Top-1 and 4.4% in Top-K | Helped |
| use Top-K patches instead of Top-1 for local similarity (ablation) | GLSim · MSCOCO | AUROC | Helped |
| select Top-K patches with Visual Logit Lens rather than attention weights or cosine similarity (ablation) | GLSim · MSCOCO | AUROC: +12.5% over attention weights and 2.6% over cosine similarity | Helped |
| use the last instruction token instead of average image-token similarity for global scoring (ablation) | GLSim · MSCOCO | AUROC: 8% | Helped |
| use an unweighted rather than probability-weighted average of local similarities (ablation) | GLSim · MSCOCO | AUROC | Helped |
| use 32 selected patches for LLaVA and 16 for Shikra (ablation) | GLSim · MSCOCO | AUROC | Mixed |
| set the global/local weighting parameter to 0.6 (ablation) | GLSim · MSCOCO | AUROC | Helped |
| use the first rather than last or average object token (ablation) | GLSim · MSCOCO | AUROC | Helped |
| use L2 distance rather than cosine similarity (ablation) | GLSim · MSCOCO | AUROC: 0.3%; 1.7% | Mixed |
| combine global and local embedding similarity | GLSim · MSCOCO | AUROC: 83.7 vs 75.4; 83.0 vs 70.7 | Helped |
| combine global and local embedding similarity | GLSim · Objects365 | AUROC | Helped |
| combine global and local embedding similarity | GLSim · MSCOCO | AUROC: 2.0%; 4.2%; 14.3%; 5.3%; 4.4% | Helped |
Surprising findings
- Attention-based grounding for top-K patch selection performs much worse than logit lens (66.3 vs 78.8 local AUROC on LLaVA), indicating high attention weights often fall on irrelevant regions.
- Global-only scoring with the last image token performs poorly (65.9 on LLaVA), while the last instruction token performs best.
Limitations
- Focuses only on object existence hallucinations; attribute and relation hallucinations are not benchmarked properly.
- Hyperparameters (layer indices, K, w) are selected on a separate validation set and vary per model.
- Attribute/relation extension relies on GPT-4o pseudo ground truth rather than human-verified labels.
Experimental setup
Datasets: MSCOCO Objects365
Benchmarks: MSCOCO Objects365
Metrics: AUROC AUPR accuracy F1 precision recall
Baselines: NLL Entropy Internal Confidence SVAR Contextual Lens POPE External model-based (LLaVA-1.5-13B judge)
Builds on: Logit Lens Internal Confidence Contextual Lens SVAR
Closest papers by meaning
- Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.92)
- PAS: Prelim Attention Score for Detecting Object Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.91)
- HalLoc: Token-level Localization of Hallucinations for Vision Language Models (CVPR 2025 · similarity 0.91)
- G2LDetect: A Global-to-Local Approach for Hallucination Detection (AAAI 2025 · similarity 0.90)
- Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.90)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.