Research map / Vision-language and multimodal / Human and embodied modeling
Coarse-to-Fine Concept Bottleneck Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coarse-to-Fine Concept Bottleneck Models (CF-CBMs), a two-level CBM where whole-image high-level concepts gate patch-level low-level attributes via amortized Bernoulli indicators trained with a variational objective, and introduces a Jaccard-based attribute-matching metric for interpretability.
Problem
Concept bottleneck models use concepts matched only to the whole image, losing fine-grained, patch-level concept information, and their decisions often rely on too many concepts to interpret.
Approach
A hierarchy in which each high-level concept is characterized by low-level attributes matched to image patches; per-example Bernoulli indicators at both levels, linked top-down, select sparse concepts end-to-end, and a Jaccard index against ground-truth attributes measures interpretability.
Key results
- CF-CBM high level reaches 79.50% on CUB with 50% active concepts, versus 80.30% for CDM without sparsity and 78.90% for CDM with sparsity at 19%.
- CF-CBM low level reaches 78.45% on ImageNet with 15% active concepts, versus 58.20% for CDM low level with sparsity.
- On attribute matching, CF-CBM improves class-wise Jaccard on CUB to 32.50% versus 17.20% for CDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| link whole-image high-level concepts to patch-level attributes | CF-CBM versus CDM H with discovery · ImageNet-1k | accuracy: 76.55 -> 77.40 | Helped |
| link whole-image high-level concepts to patch-level attributes | CF-CBM versus CDM H with discovery · CUB | accuracy: 78.90 -> 79.50 | Helped |
| link whole-image high-level concepts to patch-level attributes | CF-CBM versus CDM H with discovery · SUN | accuracy: 64.55 -> 64.00 | Hurt |
| add hierarchical linkage between high-level concepts and patch-level attributes (ablation) | patch-level CDM · CUB | accuracy: 59.62 -> 73.20 | Helped |
| add hierarchical linkage between high-level concepts and patch-level attributes (ablation) | patch-level CDM · SUN | accuracy: 42.30 -> 57.10 | Helped |
| add hierarchical linkage between high-level concepts and patch-level attributes (ablation) | patch-level CDM · ImageNet-1k | accuracy: 58.20 -> 78.45 | Helped |
| use concept hierarchy and patch-level discovery | CDM · CUB, class-wise attribute matching | Jaccard index: 17.20 -> 32.50 | Helped |
| use concept hierarchy and patch-level discovery | CDM · SUN, example-wise attribute matching | Jaccard index: 15.70 -> 16.80 | Helped |
| use 16 rather than 4 patches (ablation) | CF-CBM · CUB, class-wise attribute matching | Jaccard index: 27.20 -> 32.50 | Helped |
| use 64 rather than 16 patches (ablation) | CF-CBM · CUB, example-wise attribute matching | Jaccard index: 17.60 -> 18.00 | Helped |
| use 256 rather than 16 patches (ablation) | CF-CBM · CUB, example-wise attribute matching | Jaccard index: 17.60 -> 16.70 | Hurt |
| use 256 rather than 16 patches (ablation) | CF-CBM · CUB, class-wise attribute matching | Jaccard index: 32.50 -> 27.40 | Hurt |
Surprising findings
- Binary matching accuracy can be misleading for sparse ground-truth attributes, since predicting few concepts yields high accuracy, motivating the Jaccard metric.
Limitations
- Performance and interpretability depend on the quality of the vision-language backbone's image-concept similarities.
- Evaluated only on vision-language settings with CLIP; generalization to other domains is untested.
- Assumes a known, hand-specified hierarchy between high- and low-level concepts.
Experimental setup
Datasets: CUB SUN ImageNet-1k
Benchmarks: CUB SUN ImageNet-1k
Metrics: classification accuracy sparsity (percentage of activated concepts) Jaccard index binary matching accuracy
Baselines: Label-Free CBMs CDM CLIP Embeddings ResNet baseline
Builds on: Label-Free CBM Concept Discovery Mechanism (CDM) Stochastic Gradient Variational Bayes Gumbel-Softmax
Closest papers by meaning
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.89)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.89)
- Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models (CVPR 2025 · similarity 0.88)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.88)
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.88)
- Hyperbolic Concept Bottleneck Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.