Research map / Vision-language and multimodal / Multimodal LLMs
CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces part-focused semantic co-segmentation and CALICO, an LVLM with a DINOv2-based correspondence extraction module and lightweight adaptation modules, trained on the new MIXCROPS-style multi-image part dataset (~2.4M samples).
Problem
Existing LVLM segmentation models segment single images and struggle with cross-image reasoning at object-part granularity, identifying common and unique parts across images with labels.
Approach
Augment a pretrained segmentation LVLM (GLaMM) with a Correspondence Extraction Module that fuses DINOv2 semantic features with EVA-CLIP features via cross-attention, and Correspondence Adaptation Modules that inject instruction-conditioned, correspondence-aware visual queries into Q-Former at selected LLM layers, training only ~0.3% of parameters.
Key results
- CALICO reaches 45.9 AP50, 63.7 mIoU and 59.7 Recall on the benchmark, versus 41.4 AP50 / 59.7 mIoU / 55.5 Recall for multi-image LISA.
- Trainable parameters are about 29M (0.3% of the model), with 32 image tokens per image versus 256 for LISA and 576 for GLaMM.
- TFLOPS drop to 28.5 from 44.0 (LISA) and 42.3 (GLaMM), with average inference time of 9.1 s per sample versus 13.1 s for LISA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CALICO instead of the strongest baseline | Multi-image part-focused co-segmentation · M3CoD benchmark | mIoU: 6.3% relative improvement | Helped |
| Add Q-Former instead of using a linear projection (ablation) | CALICO · M3CoD benchmark | AP50: 38.5 -> 45.9 | Helped |
| Add DINOv2 features to CEM (ablation) | CALICO · M3CoD benchmark | mIoU: 61.7 -> 63.7 | Helped |
| Add Correspondence Extraction Module (ablation) | CALICO · M3CoD benchmark | mIoU: 61.6 -> 63.7 | Helped |
| Add Correspondence Adaptation Modules (ablation) | CALICO · M3CoD benchmark | SS: 82.0 -> 82.7 | Helped |
| Add both CEM and CAM (ablation) | CALICO · M3CoD benchmark | S-IoU: 76.3 -> 77.1 | Helped |
| Keep CAM while removing CEM rather than removing both modules (ablation) | CALICO · M3CoD benchmark | mIoU: 62.7 -> 61.6 | Hurt |
| Inject CAMs at two evenly spaced layers instead of one or three layers (ablation) | CALICO · M3CoD benchmark | segmentation and labeling performance | Helped |
| Use Q-Former to reduce image-token count | CALICO versus Multi-Image LISA · M3CoD benchmark | TFLOPS: 35.23% | Helped |
Surprising findings
- Removing the adaptation modules (w/o CAM) gives the same AP50 (45.9) as the full model, and keeping CAM without CEM performs worse than removing both modules.
Limitations
- Part segmentation is limited to pairs of images; the model's multi-image generalization beyond pairs is not evaluated.
- Benchmark built from only three source datasets and on a manually curated set of comparable object pairs, which may limit generality.
- Baselines are adapted from single-image models and rely on heuristic comparison (e.g., Multi-Image VLPart) rather than purpose-built multi-image training.
Experimental setup
Datasets: PartImageNet PACO-LVIS ADE20K-Part-234 GranD
Benchmarks: part-focused semantic co-segmentation test set (~1K image pairs from the new dataset)
Metrics: AP50 mIoU Recall Semantic Similarity Semantic IoU TFLOPS inference time
Baselines: Cascade (Sparkles + GPT-4o + LISA) Multi-Image PartGLEE Multi-Image VLPart Multi-Image GLaMM Multi-Image LISA
Builds on: LISA GLaMM VPG-C InstructBLIP
Closest papers by meaning
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.89)
- LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation (NeurIPS 2025 · similarity 0.88)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.87)
- Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025 · similarity 0.87)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
- unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.