Research map / Vision-language and multimodal / Human and embodied modeling
DCBM: Data-Efficient Visual Concept Bottleneck Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCBM, a data-efficient visual concept bottleneck model whose concepts are clustered crops from segmentation/detection foundation models on 50 images per class, named via CLIP text alignment.
Problem
Existing concept bottleneck models rely on concept sets from LLMs or large image/text corpora, which limits them in data-sparse and fine-grained settings and adds modality gaps.
Approach
Define concepts as image regions proposed by segmentation or detection foundation models (SAM, SAM2, GroundingDINO, Mask R-CNN, DETR) on a small subset of training images, cluster their CLIP embeddings with k-means, use centroids as visual concepts for a sparse linear CBM, and name them via CLIP text matching.
Key results
- DCBM-MASK-RCNN reaches 82.4% top-1 on CUB with CLIP ViT-L/14, the highest among compared CBMs, while DN-CBM leads on CIFAR-10/100 and Places365.
- On ImageNet-R, DCBM-MASK-RCNN has 44.6% error vs 55.2% for DN-CBM, with a smaller IID-to-OOD gap (22.4 vs 38.8 points).
- Using only 50 images per class for concept generation, DCBM-SAM2 reaches 77.9% on ImageNet, versus 75.3% zero-shot CLIP and 83.9% linear probe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual-region concepts from DCBM-SAM2 instead of DN-CBM concepts | CBM · ImageNet-R (IN-200 trained) | error rate: 55.2 -> 48.5 | Helped |
| Use visual-region concepts from DCBM-MASK-RCNN instead of DN-CBM concepts | CBM · CUB | top-1 accuracy: 63.5* -> 82.4 | Helped |
| Use SAM2 rather than GroundingDINO or Mask R-CNN for concept proposals | DCBM · ClimateTV | top-1 accuracy: 85.6, 81.8, 87.9 | Mixed |
| Use DCBM concepts rather than a linear probe | CLIP classifier · MiT-States | top-1 accuracy: 37.3 -> 43.3 | Helped |
| Use DCBM concepts rather than a linear probe | CLIP classifier · ClimateTV | top-1 accuracy: 84.5 -> 87.9 | Helped |
| Apply PCA with 100 components before clustering (ablation) | DCBM-SAM2 · CUB | top-1 accuracy: 81.8 -> 81.9 | No clear effect |
| Apply PCA with 100 components before clustering (ablation) | DCBM-GDINO · CUB | top-1 accuracy: 81.3 -> 81.1 | No clear effect |
| Remove the background concept (ablation) | DCBM with segmentation proposals · CUB | top-1 accuracy: SAM2 ViT-L/14: 82.21 -> 80.98; MaskRCNN ViT-L/14: 83.32 -> 83.60 | Mixed |
| Replace hierarchical clustering with K-means (ablation) | DCBM · ImageNette, ImageWoof, and CUB | top-1 accuracy: SAM2 ViT-L/14: ImageWoof 95.80 -> 95.70; CUB 82.12 -> 81.88 | Mixed |
| Use a new 50-images-per-class subset instead of the main subset (ablation) | DCBM-GDINO · ImageNet, Places365, CIFAR-10, CIFAR-100 | top-1 accuracy: 77.4 -> 77.5; 52.2 -> 52.2; 97.5 -> 97.6; 85.3 -> 85.4 | No clear effect |
| Use the combined 100-images-per-class subsets instead of one 50-images-per-class subset (ablation) | DCBM-GDINO · ImageNet | top-1 accuracy: 77.4 -> 77.1 | Hurt |
| Reduce concept-generation training images to 50 per class (ablation) | DCBM · ImageNet, Places365, CIFAR-10, CIFAR-100 | top-1 accuracy: 77.4 -> 75.5; 52.2 -> 46.1; 97.5 -> 91.2; 85.3 -> 79.1 | Hurt |
Surprising findings
- On MiT-States, zero-shot CLIP (48.1%) outperforms all DCBM variants and the linear probe (37.3%) is below DCBM.
Limitations
- Accuracy on large-scale ImageNet and Places365 is slightly below other CBMs.
- Applicability depends on the segmentation model and the expressiveness of the CLIP embedding space.
- Concept set is clustered without further steering, and spurious correlations from CLIP remain possible.
Experimental setup
Datasets: ImageNet Places365 CUB-200-2011 CIFAR-10 CIFAR-100 ImageNette ImageWoof AwA2 CelebA ImageNet-R MiT-States ClimateTV
Benchmarks: ImageNet Places365 CUB-200-2011 CIFAR-10 CIFAR-100 ImageNet-R MiT-States ClimateTV AwA2 CelebA
Metrics: top-1 accuracy error rate gap grid pointing game Gini index grid pointing game relative percentage grid pointing game max score
Baselines: LF-CBM LaBo CDM DCLIP DN-CBM linear probe zero-shot CLIP
Builds on: Label-free CBM DN-CBM LaBo CLIP
Closest papers by meaning
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.91)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.91)
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.89)
- Stochastic Concept Bottleneck Models (NeurIPS 2024 · similarity 0.89)
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.89)
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.