Research map / Label-efficient and robust learning / Recognition and long-tail

Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Concept bottleneck models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes synthetic benchmarks (robots and Sudoku generators) for concept bottleneck models covering decision support and automation, and uses them to diagnose four CBM architectures.

Problem

Concept-annotated datasets for concept bottleneck models are scarce and poorly controlled, so it is hard to know when CBMs work, what drives failures, and which algorithms perform well.

Approach

Synthetic, deterministic dataset generators with known labeling rules and controllable concept sources, annotation quality, and intervention regimes, with use-case metrics for decision support (gain from interventions over a black-box baseline) and automation (net work automated minus intervention cost).

Key results

• On robots with human_concepts and perfect interventions, CBM rises from 79.9% (k=0) to 94.9% at max interventions, while ECBM stays flat at 87.5% for partial budgets.
• On sudoku at 18px, ProbCBM automates 78% of boards without interventions versus 44-50% for CBM, CEM and ECBM at selective accuracy 0.95.
• Constraining the HasKnees frontend weight to zero keeps training accuracy at about 88% but turns CBM intervention gain from +6.9% to -8% over the DNN on human_concepts.

Surprising findings

• The DNN baseline on robots predicts the majority class (87.5%) and finds no signal beyond the class prior.
• Automated concept sources can make CBM interventions hurt accuracy (-13% to -49%) even when concepts are accurate but task-irrelevant.

Limitations

• Demonstrations are single-setting diagnostics on controlled synthetic tasks, not substitutes for deployment-data validation.
• The robot DNN baseline predicts the majority class in the main setting, so black-box comparisons are weak.
• Work metric assumes equal concept-check costs; full seed or uncertainty sweeps are outside the main text.

Experimental setup

Datasets: robots (synthetic) sudoku (synthetic) CUB-200 AwA2 CelebA SkinCon Derm7pt CheXpert OAI PH2 CEBaB DTD

Benchmarks: robots sudoku

Metrics: accuracy gain from interventions selective accuracy coverage work automated (wrk) concept leakage (CTL, ICL)

Baselines: CBM CEM ProbCBM ECBM DNN

Builds on: concept bottleneck models concept embedding models ProbCBM ECBM CLIP-Dissect

Closest papers by meaning

• A theoretical design of concept sets: improving the predictability of concept bottleneck models (NeurIPS 2024 · similarity 0.90)
• Stochastic Concept Bottleneck Models (NeurIPS 2024 · similarity 0.89)
• Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations (NeurIPS 2026 · similarity 0.88)
• Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization (ICML 2025 · similarity 0.88)
• An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotations (NeurIPS 2025 · similarity 0.88)
• Editable Concept Bottleneck Models (ICML 2025 · similarity 0.87)