Research map / Detection and segmentation / Open-set and few-shot detection
CLIM: Contrastive Language-Image Mosaic for Region Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIM, which mosaics multiple image-caption pairs into one canvas and treats each sub-image as a pseudo region aligned to its caption via contrastive loss, improving open-vocabulary detectors (Detic, BARON, F-VLM, OV-RCNN) and CLIP region representations without box annotations.
Problem
Learning region-text alignment for open-vocabulary object detection requires expensive box-level text annotations, while image-text pairs lack object location information.
Approach
Mosaic several image-text pairs into a grid canvas so each sub-image becomes a pseudo region with known box location and text; then align pseudo-region features to their captions with a contrastive loss, giving region-text supervision for free from image-text data.
Key results
- On OV-COCO, CLIM improves Detic by 5.1 AP50 novel (30.3 to 35.4) and BARON by 2.1 AP50 novel (34.8 to 36.9).
- On OV-LVIS, CLIM improves F-VLM (RN50x64) by 2.2 mAP rare mask (30.1 to 32.3) and Detic by 2.3 mAP rare (19.5 to 21.8).
- In a sanity check, increasing caption batch size from 4 to about 20 yields only 3.0 AP50 novel gain versus 7.9 gain from CLIM (24.4 to 32.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLIM to caption supervision | Detic · OV-COCO | AP50 novel: 30.3 -> 35.4; +5.1 | Helped |
| Apply CLIM to caption supervision | BARON · OV-COCO | AP50 novel: 34.8 -> 36.9; +2.1 | Helped |
| Apply CLIM during vision-language pre-training | OV-RCNN · OV-COCO | AP50 novel: 22.8 -> 26.2; +3.4 | Helped |
| Train CLIP with region-text alignment | CLIP ViT-B-16 · OV-LVIS | mAP rare mask: 14.8 -> 20.8 | Helped |
| Increase caption batch size from 4 to 20 without CLIM (ablation) | Detic · OV-COCO | AP50 novel: 24.4 -> 27.4 | Helped |
| Use CLIM instead of simply increasing the caption batch size (ablation) | Detic · OV-COCO | AP50 novel: 24.4 -> 32.3; 7.9 AP50 | Helped |
| Increase caption batch size from 20 to 32 without CLIM (ablation) | Detic · OV-COCO | AP50 novel: 27.4 -> 26.3 | No clear effect |
| Randomly vary mosaic grid sizes instead of fixing one pattern (ablation) | Detic + CLIM · OV-COCO | AP50 novel: 31.2 -> 32.3; +1.1 | Helped |
| Group pseudo regions by text similarity instead of sampling randomly (ablation) | Detic + CLIM · OV-COCO | AP50 novel: 32.3 -> 31.3 | Hurt |
| Group pseudo regions by image similarity instead of sampling randomly (ablation) | Detic + CLIM · OV-COCO | AP50 novel: 32.3 -> 31.6 | Hurt |
| Add image tag loss to Detic without CLIM (ablation) | Detic · OV-COCO | AP50 novel: 24.4 -> 30.3 | Helped |
| Add image tag loss to Detic + CLIM (ablation) | Detic + CLIM · OV-COCO | AP50 novel: 32.3 -> 35.4 | Helped |
Surprising findings
- Simply increasing the number of images in the caption batch yields much smaller gains than CLIM, indicating the gain is not just from more images.
- Grouping pseudo regions by text or image similarity decreased performance relative to random mosaicking.
Limitations
- Scaling input resolution for mosaicked images is limited by quadratic-to-quartic computation cost.
- Gains depend on the baseline; RegionCLIP comparison relies on its RPN and noun vocabulary assumptions.
- Mosaicking differs from natural scene statistics, so pseudo regions may not fully match real object distributions.
Experimental setup
Datasets: COCO Caption CC3M COCO LVIS PASCAL Context
Benchmarks: OV-COCO OV-LVIS PASCAL Context
Metrics: AP50 novel AP50 base mAP rare mask Top-1 region classification accuracy Top-5 region classification accuracy
Baselines: Detic BARON RegionCLIP F-VLM OV-RCNN VLDet ViLD OADP
Builds on: Detic BARON F-VLM OV-RCNN MaskCLIP CLIP
Closest papers by meaning
- Contrastive Localized Language-Image Pre-Training (ICML 2025 · similarity 0.88)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.87)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.87)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.87)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.