Research map / Vision-language and multimodal / Human and embodied modeling
VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLG-CBM, which grounds LLM-generated concepts with Grounding DINO bounding boxes to train a concept bottleneck layer, and introduces a Number of Effective Concepts (NEC) metric to control information leakage.
Problem
Concept bottleneck models trained with LLM-generated concepts and VLM supervision suffer from concept predictions that mismatch images and from information leakage, where random concept layers reach high accuracy.
Approach
Ground candidate concepts on training images with an open-vocabulary detector (Grounding DINO Swin-B) to create concept-labeled data for training the CBL, and evaluate CBMs by accuracy at a fixed number of effective concepts (NEC) enforced via a sparse final layer, supported by a theorem that random CBLs approximate linear classifiers as concept count grows.
Key results
- With CLIP-RN50, VLG-CBM reaches 88.55% ANEC-5 on CIFAR10 and 59.74% on ImageNet, versus 84.05% and 52.88% for LF-CBM.
- On CUB with non-CLIP ResNet-18, VLG-CBM reaches 75.79% ANEC-5 versus 53.51% for LF-CBM; on Places365 it reaches 41.92% versus 37.65%.
- Pruning to top-5 concepts changes 0.12% of VLG-CBM predictions versus 49.21% (LF-CBM), 98.34% (LM4CV), and 81.40% (LaBo).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VLG-CBM instead of the strongest reported baseline | CBMs with CLIP-RN50 · CIFAR10, CIFAR100, ImageNet, and CUB | ANEC-5: at least 4.27% and up to 51.09% | Helped |
| use VLG-CBM instead of the strongest reported baseline | CBMs with CLIP-RN50 · CIFAR10, CIFAR100, ImageNet, and CUB | ANEC-avg: at least 0.45% and up to 29.78% | Helped |
| use VLG-CBM instead of LF-CBM | CBMs with non-CLIP backbones · CUB, Places365, and ImageNet | ANEC-5 | Helped |
| set confidence threshold T to 0.10 rather than 0.20 (ablation) | VLG-CBM · CUB200 | ANEC-5: 75.73% -> 75.75% | Helped |
| set confidence threshold T to 0.10 rather than 0.20 (ablation) | VLG-CBM · Places365 | ANEC-5: 41.25% -> 41.84% | Helped |
| set crop-to-concept augmentation probability to 0.2 rather than 0.0 (ablation) | VLG-CBM · CUB | ANEC-5: 75.73 -> 75.83 | Helped |
| set crop-to-concept augmentation probability to 0.4 rather than 0.2 (ablation) | VLG-CBM · CUB | ANEC-5: 75.83 -> 75.71 | Hurt |
| set crop-to-concept augmentation probability to 0.6 rather than 0.2 (ablation) | VLG-CBM · CUB | ANEC-5: 75.83 -> 75.57 | Hurt |
| set crop-to-concept augmentation probability to 0.8 rather than 0.2 (ablation) | VLG-CBM · CUB | ANEC-5: 75.83 -> 75.52 | Hurt |
| set crop-to-concept augmentation probability to 1.0 rather than 0.2 (ablation) | VLG-CBM · CUB | ANEC-5: 75.83 -> 72.29 | Hurt |
Surprising findings
- A random concept bottleneck layer can approach the accuracy of trained CBMs when the number of concepts is large, supporting information leakage.
- LM4CV performs worse than a random CBL at small NEC.
Limitations
- Depends on large pretrained models such as Grounding DINO and CLIP.
- Grounding DINO annotations have relatively low precision (about 0.72) though high recall.
- Concept-labeled data is generated only for image classification benchmarks.
Experimental setup
Datasets: CIFAR10 CIFAR100 CUB Places365 ImageNet Waterbirds
Benchmarks: CIFAR10 CIFAR100 CUB Places365 ImageNet Waterbirds
Metrics: ANEC-5 ANEC-avg accuracy precision recall percent of changed decisions after top-5 pruning
Baselines: LF-CBM LaBo LM4CV Random CBL
Builds on: Label-free CBM (LF-CBM) GLM-SAGA Grounding DINO
Closest papers by meaning
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.94)
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.93)
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.91)
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.91)
- Bayesian Concept Bottleneck Models with LLM Priors (NeurIPS 2025 · similarity 0.91)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.