Research map / Detection and segmentation / Open-set and few-shot detection
Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProMaC, a training-free test-time iterative prompt-mask cycle that uses MLLM hallucinations (multi-scale chain-of-thought plus visual contrastive reasoning with inpainted contrastive images) and SAM mask semantic alignment to produce instance-specific prompts for task-generic promptable segmentation.
Problem
Task-generic promptable segmentation needs instance-specific prompts derived from a single task-level prompt, but MLLMs hallucinate (from co-occurrence priors) when producing these prompts, degrading segmentation accuracy.
Approach
Treat MLLM hallucinations as useful contextual prior knowledge rather than errors: expand candidate prompts via multi-scale image patches, then verify and prune them with visual contrastive reasoning on inpainted images (inpainting out SAM-predicted masks), and feed the aligned masks back to the prompt generator in an iterative cycle.
Key results
- On COD10K/CHAMELEON/CAMO, ProMaC reaches M 0.042/0.044/0.090 and S-measure 0.805/0.833/0.767, outperforming task-generic baselines such as GenSAM (0.783 S on COD10K).
- On CVC-ColonDB polyp segmentation, ProMaC reduces MAE to 0.176 versus 0.244 for GenSAM; on ISIC, F-beta is 0.717 vs 0.699 for GenSAM.
- On open-vocabulary segmentation, ProMaC achieves 59.3 mIoU on PASCAL VOC, above the next best of 53.8 (OVSegmentor).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · CHAMELEON | M: 0.073 -> 0.044 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · CAMO | M: 0.106 -> 0.090 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · COD10K | Fβ: 0.695 -> 0.716 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · CVC-ColonDB | Fβ: 0.059 -> 0.243 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · Kvasir | Fβ: 0.210 -> 0.394 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · ISIC | Fβ: 0.699 -> 0.717 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · GSD | Fβ: 0.394 -> 0.409 | Helped |
| Use ProMaC instead of GenSAM | Task-generic promptable segmentation · Trans10K-hard | Fβ: 0.489 -> 0.509 | Helped |
| Use ProMaC instead of TCL | Open-vocabulary segmentation · COCO-Object val | mIoU: 30.4 -> 25.2 | Hurt |
| Add VCR to LLaVA-1.5-13B | LLaVA-1.5-13B · What’s Up | accuracy: 73.1 -> 75.4 | Helped |
| Use multi-scale chain-of-thought patches instead of only the original image (ablation) | ProMaC · CHAMELEON | Fβ: 0.764 -> 0.790 | Helped |
| Use multimodal text and visual prompts instead of text-only prompts (ablation) | ProMaC · CHAMELEON | Fβ: 0.685 -> 0.790 | Helped |
Surprising findings
- Multiple iterations degraded performance for the GPT4V+SAM and LLaVA1.5+SAM baselines, so they were run for a single iteration only.
- Bounding-box visual markers decreased LLaVA performance, whereas the proposed inpainting-based contrastive samples improved it.
Limitations
- Performance on specialized medical data is limited by the generalization of the underlying MLLM (LLaVA), falling short of COD results.
- Relies on the MLLM's prior knowledge, so hallucinations can still mislead prompts when the MLLM lacks domain data.
- Iterative test-time adaptation adds compute cost per image (4-6 epochs with inpainting and multiple MLLM calls).
Experimental setup
Datasets: CHAMELEON CAMO COD10K CVC-ColonDB Kvasir ISIC GSD Trans10K-hard PASCAL VOC Pascal Context COCO-Object What's Up
Benchmarks: CHAMELEON CAMO COD10K CVC-ColonDB Kvasir ISIC GSD Trans10K-hard PASCAL VOC Pascal Context COCO-Object What's Up
Metrics: M (MAE) F-beta E-measure S-measure mIoU accuracy
Baselines: GenSAM GPT4V+SAM LLaVA1.5+SAM GroundingSAM SEEM X-Decoder WS-SAM SAM-P CLIP_Surgery+SAM MaskCLIP GroupViT
Builds on: SAM GenSAM LLaVA-1.5 classifier-free guidance visual contrastive decoding
Closest papers by meaning
- Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.90)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.89)
- Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.88)
- Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection (CVPR 2026 · similarity 0.88)
- Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.88)
- Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.