Research map / Probabilistic and scientific modeling / Molecules and proteins
Stochastic Concept Bottleneck Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stochastic Concept Bottleneck Models (SCBMs), which model concept logits with a learned multivariate normal distribution to capture concept dependencies so single interventions propagate to correlated concepts, plus a confidence-region-based intervention strategy.
Problem
Concept bottleneck models treat concepts as independent, so user interventions on one concept do not propagate to correlated concepts, making interventions laborious and less effective.
Approach
Parameterize the concept bottleneck with a non-diagonal multivariate normal over concept logits (amortized or global covariance), train end-to-end with Monte Carlo likelihood and Gumbel-Softmax hard concepts, and choose intervention logits via a likelihood-based confidence region that respects concept correlations.
Key results
- On CUB, Amortized SCBM reaches 69.87% target accuracy prior to interventions versus 67.72% for Hard CBM, with concept accuracy 95.22% (comparable to baselines).
- Intervention curves with uncertainty-based policy are steeper for SCBMs than for Hard CBM, CEM, and Autoregressive CBM on synthetic, CUB, and CIFAR-10, especially for few interventions.
- Autoregressive CBM has about 15x slower inference on CUB, while SCBMs match Hard CBM at 1x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model concept dependencies with a multivariate normal distribution and use dependency-aware interventions | SCBM · Synthetic, CUB, CIFAR-10; uncertainty-ordered interventions | target accuracy | Helped |
| Use amortized rather than global covariance (ablation) | SCBM · CUB and CIFAR-10; uncertainty-ordered interventions | target accuracy | Helped |
| Use global rather than amortized covariance (ablation) | SCBM · Synthetic; uncertainty-ordered interventions | target accuracy | Helped |
| Select concepts by predicted uncertainty rather than randomly (ablation) | SCBM and baseline methods · CUB; intervention policy | target accuracy | Helped |
| Set intervention logits using the confidence-region strategy rather than training-distribution percentiles (ablation) | SCBM · CUB; uncertainty-ordered interventions | target accuracy | Helped |
| Vary the covariance regularization strength λ₂ (ablation) | SCBM · CUB; uncertainty-ordered interventions | target accuracy | No clear effect |
| Reduce Monte Carlo samples to M=10 (ablation) | SCBM · Intervention performance | target accuracy | No clear effect |
| Vary the confidence-region level (ablation) | SCBM · CUB; uncertainty-ordered interventions | target accuracy | No clear effect |
Surprising findings
- CEM has the worst concept accuracy on CIFAR-10 yet the best target accuracy, suggesting possible leakage in its embeddings.
- Autoregressive CBM shows a negative dip in intervention performance on CIFAR-100, attributed to independent target predictor training.
Limitations
- Limited to binary concepts; extension to continuous concepts left for future work.
- Covariance matrix has quadratic memory complexity in the number of concepts, limiting scaling to large concept sets.
- Intervention strategy models only model uncertainty, not user uncertainty or imperfect human interventions.
Experimental setup
Datasets: Synthetic tabular (custom) CUB-200-2011 CIFAR-10 CIFAR-100
Benchmarks: Synthetic tabular CUB-200-2011 CIFAR-10 CIFAR-100
Metrics: concept accuracy target accuracy concept Jaccard index Brier score expected calibration error
Baselines: Hard CBM Concept Embedding Model (CEM) Autoregressive CBM
Builds on: Concept Bottleneck Models Concept Embedding Models Stochastic Segmentation Networks Graphical Lasso Gumbel-Softmax
Closest papers by meaning
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.92)
- Learning to Intervene on Concept Bottlenecks (ICML 2024 · similarity 0.91)
- Bayesian Concept Bottleneck Models with LLM Priors (NeurIPS 2025 · similarity 0.91)
- Editable Concept Bottleneck Models (ICML 2025 · similarity 0.90)
- Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions (NeurIPS 2025 · similarity 0.90)
- An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotations (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.