Research map / Vision-language and multimodal / Multimodal LLMs
CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPCF, a cross-prompt contrastive decoding framework for referring MLLMs that contrasts the target-region prompt with auto-extracted contrastive prompts from misleading regions, plus self-training and distillation to cut inference cost.
Problem
Referring MLLMs often produce incorrect responses by attending to misleading regions adjacent to or similar to the referred target region.
Approach
Contrast outputs conditioned on the true visual prompt against outputs conditioned on contrastive point prompts from misleading regions, with a learned prompt extraction network (instruction-dependent, prior-informed) to select those prompts, self-training via RAG-generated QA with DPO, and distillation into a single-pass adapter-augmented model.
Key results
- CPCF-7B reaches 78.37% on ROC-Box vs 71.71% for Ferret-7B and 79.09% for CPCF-13B on the same task.
- CPCF-7B scores 83.5 on Ferret-Bench referring captioning vs 68.7 for Ferret-7B and 79.9 for Ferret-v2-7B.
- Distillation to a single-execution model keeps performance near the multi-execution version (79.08 vs 78.37 on ROC-Box) at 26.4% of its inference time, with only 4.3% overhead over Ferret.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cross-prompt contrastive decoding with automatically extracted prompts | Ferret-7B · ROC Box | accuracy: 75.40 -> 78.37 | Helped |
| use cross-prompt contrastive decoding with automatically extracted prompts | Ferret-7B · ROC Mask | accuracy: 76.20 -> 79.55 | Helped |
| use cross-prompt contrastive decoding with automatically extracted prompts | Ferret-7B · ROC Scribble | accuracy: 75.24 -> 77.97 | Helped |
| use cross-prompt contrastive decoding with automatically extracted prompts | Ferret-7B · ROC Point | accuracy: 73.39 -> 77.14 | Helped |
| replace automatic prompt extraction with a prior-based manual prompt method (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 74.33 | Hurt |
| remove self-training and train the prompt extraction network directly on D (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 73.19 | Hurt |
| remove distillation and use multi-execution inference (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 79.08 | Helped |
| remove BLIP2 Q-Former initialization (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 76.93 | Hurt |
| remove semantic similarity map Ms (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 76.50 | Hurt |
| remove relative distance map Md (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 75.77 | Hurt |
| remove random Gaussian noise from prompt selection (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 77.51 | Hurt |
| generate questions without RAG (ablation) | CPCF-7B · ROC Box | accuracy: 78.37 -> 75.98 | Hurt |
Surprising findings
- Removing distillation slightly improves accuracy (79.08 vs 78.37 on ROC-Box) but raises inference time by more than threefold.
Limitations
- Contrastive decoding with multiple prompts still costs extra compute in the multi-execution version.
- Prompt extraction network and self-training depend on Ferret-style pretraining and a RAG pipeline that relies on the MLLM's own generated descriptions.
Experimental setup
Datasets: Ferret training data Open Images (50K images) LVIS validation COCO-Text RefCOCOg Ferret-Bench
Benchmarks: ROC (LVIS-based, from Wu et al. 2024) RTC (COCO-Text-based) RefCOCOg referring description Ferret-Bench referring reasoning and captioning
Metrics: accuracy BLEU-4 METEOR CIDEr SPICE GPT-based score
Baselines: Kosmos2 GPT4RoI Shikra CogVLM GLaMM Osprey Ferret Ferret-v2 VCD ICD CRG
Builds on: Ferret contrastive decoding (VCD) CRG DPO BLIP-2 Q-Former
Closest papers by meaning
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.87)
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.86)
- VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.