Research map / Vision-language and multimodal / Human and embodied modeling
LocCa: Visual Pretraining with Location-aware Captioners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LocCa, a captioner-based visual pretraining method that adds location-aware tasks (automatic referring expressions and grounded captioning) to an encoder-decoder captioner, yielding SOTA on RefCOCO-family localization while staying competitive on holistic tasks.
Problem
Image captioning pretraining captures holistic image content but overlooks region-specific detail; contrastive and RoI-based methods for localization need complex architectures or are computationally demanding.
Approach
Use natural language as the interface for visual pretraining: an autoregressive encoder-decoder captioner is trained from scratch with task-prefixed location-aware tasks (AREF: caption-to-box, GCAP: box-to-caption) on pseudo box annotations from OWL-ViT, with losses on both text and coordinates, so the vision encoder learns location sensitivity with no extra architecture.
Key results
- On RefCOCO clean-split evaluation (no val/test images in training), LocCa_G reaches 91.20 val / 93.34 testA on RefCOCO and state-of-the-art on the group of models that never saw val/test images.
- With a LiT-Decoder frozen encoder on RefCOCO REC, LocCa_L reaches 88.34 val versus 64.17 for CapPa, a large gain in localization.
- Integrated in PaLI-3 at L scale, LocCa outperforms SigLIP_L on COCO captioning (138.9 vs 135.8) and VQAv2 (77.6 vs 75.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AREF and GCAP location-aware pretraining tasks (ablation) | CapPa-style image captioner · RefCOCO val | Acc@0.5: 75.2 -> 88.0 | Helped |
| add GCAP pretraining (ablation) | LocCa with AREF · RefCOCO val | Acc@0.5: 86.8 -> 88.0 | Helped |
| add AREF pretraining (ablation) | LocCa with GCAP · RefCOCO val | Acc@0.5: 83.7 -> 88.0 | Helped |
| add AREF and GCAP location-aware pretraining tasks (ablation) | CapPa-style image captioner · ImageNet-1k | accuracy: 78.3 -> 80.4 | Helped |
| add AREF and GCAP location-aware pretraining tasks (ablation) | CapPa-style image captioner · COCO | CIDEr: 111.0 -> 117.7 | Helped |
| add AREF and GCAP location-aware pretraining tasks (ablation) | CapPa-style image captioner · VQAv2 | accuracy: 66.2 -> 68.4 | Helped |
| add AREF and GCAP location-aware pretraining tasks (ablation) | CapPa-style image captioner · GQA | accuracy: 54.3 -> 59.0 | Helped |
| use LocCa rather than CapPa | frozen image encoder with LiT decoder · ImageNet-1k | accuracy: 84.4 -> 84.5 | No clear effect |
| use LocCa rather than CapPa | frozen image encoder with LiT decoder · VQAv2 | accuracy: 70.9 -> 72.8 | Helped |
| use LocCa rather than CapPa | frozen image encoder with LiT decoder · GQA | accuracy: 58.3 -> 61.8 | Helped |
| use LocCa rather than CapPa | frozen image encoder with LiT decoder · COCO captioning | CIDEr: 125.8 -> 127.1 | Helped |
| use LocCa rather than CapPa | frozen image encoder with LiT decoder · Oxford-Pet | accuracy: 95.6 -> 95.0 | Hurt |
Surprising findings
- Removing all val/test images from training changes RefCOCO results; about half of RefCOCO test images overlap with combined training sets, a common but undisclosed leakage.
Limitations
- No zero-shot segmentation because there is no pixel-level pretraining.
- Zero-shot multi-object detection from raw decoder is noisy and requires beam search or finetuning.
- Grounded captioning CIDEr lags GLaMM on Visual Genome.
Experimental setup
Datasets: WebLI RefCOCO RefCOCO+ RefCOCOg COCO Objects365 ADE20k Visual Genome OpenImages
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ImageNet COCO captioning Flickr30k VQAv2 GQA OCR-VQA ADE20k COCO detection TextVQA
Metrics: Acc@0.5 mIoU CIDEr accuracy mAP mIoU (segmentation) METEOR
Baselines: CLIP* Cap CapPa SigLIP UNINEXT OFA PixelLLM Shikra Ferret
Builds on: CapPa LiT-Decoder PaLI-3 OWL-ViT Segmenter
Closest papers by meaning
- Pixel-Aligned Language Model (CVPR 2024 · similarity 0.87)
- Contrastive Localized Language-Image Pre-Training (ICML 2025 · similarity 0.87)
- Enhancing Vision-Language Pre-training with Rich Supervisions (CVPR 2024 · similarity 0.87)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.86)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.86)
- TIPS: Text-Image Pretraining with Spatial awareness (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.