Research map / Vision-language and multimodal / Human and embodied modeling
Hybrid Concept Bottleneck Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HybridCBM, a CLIP-based concept bottleneck model that combines LLM-generated static concepts with learnable dynamic concept vectors that are labeled by a pre-trained GPT-2 concept translator.
Problem
Concept Bottleneck Models depend on predefined concepts that are often incomplete, and manual concept annotation is costly, which hurts accuracy and interpretability.
Approach
Combine a static concept bank (GPT-3.5 concepts embedded by CLIP text encoder) with a dynamic concept bank of learnable vectors trained with discriminability, orthogonality and Sinkhorn distribution-alignment losses, and translate learned vectors into text with a GPT-2 prefix-LM trained to invert CLIP's text encoder.
Key results
- HybridCBM (submodular) averages 86.83% on full-data classification, nearly matching linear probe at 86.84%, and 75.12% average over shots versus 72.59% for linear probe.
- On CIFAR-10 HybridCBM reaches 88.03% and on CIFAR-100 68.38%, exceeding Labo (87.6 / 65.2) and ResCBM (88.0 / 67.9) in the reported comparison.
- Dynamic concepts reach 46.2% precision@t versus 47.3% for static concepts, showing learned concepts approach static concept interpretability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HybridCBM instead of Linear Probe | CLIP-ViT-L/14 · Mean accuracy across all datasets, 1 shot | accuracy: 52.30 -> 58.64 | Helped |
| Use HybridCBM instead of Linear Probe | CLIP-ViT-L/14 · Mean accuracy across all datasets, full | accuracy: 86.84 -> 86.83 | No clear effect |
| Use HybridCBM instead of Linear Probe | CLIP-RN50 · CIFAR-10 | accuracy: 88.8 -> 88.03 | Hurt |
| Use HybridCBM instead of Linear Probe | CLIP-RN50 · CIFAR-100 | accuracy: 70.1 -> 68.38 | Hurt |
| Add dynamic concepts to the static concept bank (ratio 0 to 0.2) (ablation) | HybridCBM (submodular), 10 concepts per class · Average across 10 datasets | accuracy: 84.27 -> 85.47 | Helped |
| Add discriminability loss (ablation) | HybridCBM (submodular) · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | accuracy: 84.60 -> 84.36 | Hurt |
| Add discriminability loss (ablation) | HybridCBM (submodular) · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | precision@t: 0.047 -> 34.8 | Helped |
| Add orthogonality loss (ablation) | HybridCBM (submodular), with Lcls+dis · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | accuracy: 84.36 -> 84.29 | Hurt |
| Add orthogonality loss (ablation) | HybridCBM (submodular), with Lcls+dis · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | precision@t: 34.8 -> 34.1 | Hurt |
| Add distribution alignment loss (ablation) | HybridCBM (submodular), with Lcls+dis+ort · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | accuracy: 84.29 -> 84.54 | Helped |
| Add distribution alignment loss (ablation) | HybridCBM (submodular), with Lcls+dis+ort · Average across 10 datasets, dynamic ratio 0.5, 10 concepts per class | precision@t: 34.1 -> 46.2 | Helped |
Surprising findings
- Random static concept selection underperforms submodular selection, and CaptionCBM (translated captions as concepts) performs worse than HybridCBM on CIFAR-10 and CIFAR-100.
Limitations
- Dynamic concept semantic scores (about 30-33%) remain well below static concepts (38.3%) and the evaluation depends on GPT-3.5/GPT-4o judgments with acknowledged LLM limitations.
- Adding interpretability losses slightly reduces classification accuracy (e.g., 84.60 to 84.29).
Experimental setup
Datasets: MSCOCO ConceptNet ImageNet CIFAR-10 CIFAR-100 Food-101 FGVC-Aircraft Flower-102 CUB-200-2011 UCF-101 DTD HAM10000
Benchmarks: ImageNet CIFAR-10 CIFAR-100 Food-101 FGVC-Aircraft Flower-102 CUB-200-2011 UCF-101 DTD HAM10000 RESISC45
Metrics: accuracy concept purity concept separation semantic alignment precision@t
Baselines: Linear Probe PCBM PCBM-h Label-Free CBM Labo ResCBM CaptionCBM
Builds on: Concept Bottleneck Models Label-Free CBM LaBo PCBM ResCBM
Closest papers by meaning
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.92)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.91)
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.91)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.91)
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.91)
- Bayesian Concept Bottleneck Models with LLM Priors (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.