Research map / Vision-language and multimodal / CLIP and zero-shot
PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a small frozen-VLM-compatible positional insert (PIN), trained with next-token loss on synthetic pasted-object data, to unlock zero-shot object localisation in OpenFlamingo and BLIP-2.
Problem
Caption-based vision-language models cannot localise objects, because their image-caption pretraining gives no explicit spatial grounding, and existing localisation fixes need supervised detection data or heavy fine-tuning.
Approach
A learnable, input-agnostic spatial prompt built from sinusoidal positional embeddings passed through a small MLP and added to the vision encoder output, trained only with next-token cross-entropy to emit bounding box text on synthetic composited images, while all VLM weights stay frozen.
Key results
- PIN on OpenFlamingo reaches 0.45 mIoU on PVOC (<=3 objects), roughly 2x the best in-context learning baseline, and 0.35 mIoU on COCO, roughly 3x.
- PIN reaches 26.4 P@0.3 on RefCOCO Test-A zero-shot versus 3.7 for in-context OpenFlamingo.
- PIN has about 1.2M parameters (0.04% of the 3B VLM) and slightly outperforms LoRA on the vision encoder for localisation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PIN to OpenFlamingo | OpenFlamingo · PVOC, COCO and LVIS (≤3 Objects) | mIoU: PVOC 0.44 -> 0.45; COCO 0.33 -> 0.35; LVIS 0.23 -> 0.26 | Helped |
| add PIN to BLIP-2 | BLIP-2 · PVOC, COCO and LVIS (≤3 Objects) | mIoU: PVOC 0.32 -> 0.44; COCO 0.26 -> 0.34; LVIS 0.17 -> 0.26 | Helped |
| add positional-referral expressions to synthetic training data (ablation) | PIN with OpenFlamingo · RefCOCO Test-A | P@0.3: 14.1 -> 26.4 | Helped |
| train PIN on COCO data instead of synthetic data (ablation) | PIN with OpenFlamingo · COCO≤3 Objects | mIoU: 0.35 -> 0.39 | Helped |
| train PIN on COCO data instead of synthetic data (ablation) | PIN with OpenFlamingo · PVOC≤3 Objects | mIoU: 0.45 -> 0.45 | No clear effect |
| increase image resolution from 224 to 448 (ablation) | PIN with OpenFlamingo · PVOC and COCO≤3 Objects | mIoU: PVOC 0.45 -> 0.47; COCO 0.35 -> 0.37 | Helped |
| increase image resolution from 224 to 364 (ablation) | PIN with BLIP-2 · PVOC and COCO≤3 Objects | mIoU: PVOC 0.44 -> 0.47; COCO 0.34 -> 0.37 | Helped |
| add PIN (ablation) | OpenFlamingo · VQAv2 | accuracy: 44.1% -> 34.3\ % | Hurt |
Surprising findings
- OpenFlamingo in-context learning performs no better than randomly selected bounding boxes, suggesting it cannot exploit positional information from context boxes.
- Fine-tuning the vision encoder with LoRA nearly perfectly solves the synthetic training examples, suggesting overfitting to synthetic artifacts.
Limitations
- Localisation is imprecise for small objects and crowded or overlapping scenes.
- Grounding on complex, longer referring expressions is limited by the small 1B LLM in OpenFlamingo.
- VQAv2 accuracy drops from 44.1% to 34.3% when PIN is inserted.
Experimental setup
Datasets: LVIS BG20k COCO PASCAL VOC RefCOCO
Benchmarks: PASCAL VOC 2007 COCO LVIS RefCOCO Test-A VQAv2
Metrics: mean IoU P@0.3 VQAv2 accuracy
Baselines: OpenFlamingo in-context learning CoOp VPT LoRA random bounding box
Builds on: Flamingo X-Paste Visual Prompt Tuning CoOp LoRA
Closest papers by meaning
- Grounding Everything in Tokens for Multimodal Large Language Models (CVPR 2026 · similarity 0.85)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.85)
- HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.85)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.85)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.