Research map / Large language models / Language, safety and interpretability
LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-wrapper, a black-box adaptation method that lightly fine-tunes an LLM to pick the best referring-expression box from text-serialized candidate boxes produced by a frozen VLM.
Problem
Zero-shot vision-language models lag behind task-specific fine-tuned models on referring expression comprehension (REC), but white-box fine-tuning is infeasible for closed-source or private VLMs.
Approach
Serialize the outputs of a frozen black-box VLM (box coordinates, labels, scores) into a text prompt and LoRA fine-tune an LLM to select the index of the best-matching box, so adaptation needs only forward calls to the VLM and transfers across VLMs, datasets, and ensembles.
Key results
- On RefCOCOg, LLM-wrapper improves zero-shot Grounding-DINO (T) by up to +18.0 P@1 (test-umd 77.36 vs 59.32) with Llama 3 8B.
- On Talk2Car and RefCOCOg, gains range from +8.7 to +18.3 P@1 across VLM/LLM combinations.
- Transferring a wrapper trained on Florence-2 outputs to Grounding-DINO yields +5.1 to +6.9 P@1 over zero-shot VLMs on RefCOCOg; a wrapper trained on GDrec applied to GD-1.5 API outputs gains +29.0 P@1 on a 300-sample subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt Grounding-DINO outputs with a fine-tuned LLM-wrapper | Grounding-DINO on RefCOCOg test-umd · RefCOCOg test-umd | P@1: +17.7 | Helped |
| Adapt Grounding-DINO outputs with a fine-tuned LLM-wrapper | Grounding-DINO on RefCOCO test-unc · RefCOCO test-unc | P@1: +22.5 | Helped |
| Adapt Grounding-DINO outputs with a fine-tuned LLM-wrapper | Grounding-DINO on RefCOCO+ test-unc · RefCOCO+ test-unc | P@1: +8.3 | Helped |
| Adapt Grounding-DINO outputs with a fine-tuned LLM-wrapper | Grounding-DINO on Talk2Car test · Talk2Car test | P@1: +8.7 | Helped |
| Ensemble Florence-2 and Grounding-DINO-REC candidate outputs | LLM-wrapper on RefCOCOg · RefCOCOg val-umd and test-umd | P@1: +2.5 P@1 (val-umd), +2.1 P@1 (test-umd) | Helped |
| Transfer the wrapper trained on Grounding-DINO-REC outputs to GD-1.5 outputs | LLM-wrapper on GD-1.5 · RefCOCOg val subset (300 samples) | P@1: +29.0 | Helped |
| Train the wrapper on RefCOCOg and transfer it to HC-RefLoCo | Florence-2 with LLM-wrapper · HC-RefLoCo | P@1: +18.9 (val), +19.1 (test) | Helped |
| Use a larger, more capable LLM | LLM-wrapper on RefCOCOg · RefCOCOg | P@1: +2.2 P@1 (GPT-Neo 139M), +18.0 P@1 (Llama 3 8B) | Helped |
| Vary LoRA rank from 12 to 192 (ablation) | LLM-wrapper with Grounding-DINO on RefCOCOg · RefCOCOg val-umd | P@1: 77.84; 77.72; 78.12; 77.70 | No clear effect |
| Omit LLM fine-tuning and use a zero-shot LLM (ablation) | Grounding-DINO with LLM-wrapper · RefCOCOg val-umd and test-umd | P@1: 60.09 -> 58.05; 59.32 -> 58.47 | Hurt |
| Train the wrapper on Florence-2 outputs and transfer it to Grounding-DINO-REC outputs | LLM-wrapper on RefCOCOg · RefCOCOg val/test | P@1: +6.3 (val), +5.1 (test) | Helped |
| Train the wrapper on Grounding-DINO-REC outputs and transfer it to Florence-2 outputs | LLM-wrapper on RefCOCOg · RefCOCOg val/test | P@1: +6.0 (val), +6.9 (test) | Helped |
Surprising findings
- Zero-shot LLM wrapping (no LLM fine-tuning) gives almost no gain over the VLM alone in most settings.
- LLM gains correlate strongly with the LLM's HellaSwag score (Pearson 0.88), while the number of trainable LoRA parameters shows near-zero correlation (-0.07).
Limitations
- Adds LLM inference cost on top of each VLM call.
- Performance is bounded by the quality and recall of the underlying VLM's candidate boxes and labels.
- Box coordinates and labels alone can be insufficient for scene understanding, e.g., for orientation or fine-grained spatial cues.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Talk2Car HC-RefLoCo
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Talk2Car HC-RefLoCo
Metrics: P@1 (IoU>0.5) cIoU gIoU
Baselines: Grounding-DINO zero-shot Florence-2 zero-shot white-box fine-tuned Grounding-DINO white-box fine-tuned Florence-2 Contrastive Region Guidance (CRG) Kosmos-2
Builds on: LoRA Grounding-DINO Florence-2 SpaCy dependency parser
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.89)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.89)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.