Research map / Vision-language and multimodal / Human and embodied modeling

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Concept bottleneck models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VH-CBM, a hybrid concept bottleneck model that propagates a small number of expert concept annotations through Gaussian Processes in a VLM (CLIP/DINO) embedding space, improving concept accuracy and calibration over VLM-guided CBMs.

Problem

Concept bottleneck models need concept-level annotations that are scarce, while VLM-guided CBMs replace them with unreliable VLM annotations that reduce concept accuracy and interpretability.

Approach

Fit per-concept multi-class GP classifiers (Dirichlet-transformed regression, latent GPs with mixing matrix, sparse variational ELBO) on CLIP or DINO embeddings using a small set of expert-annotated samples, then feed the GP concept statistics to a linear task layer; optionally select concepts to annotate by GP uncertainty.

Key results

• On CUB with 7% of concepts annotated, VH-CBM (DINO) reaches concept F1 of 0.89 and task F1 of 0.88, versus 0.79 for VLG-CBM on concept F1.
• VH-CBM (CLIP) reduces ECCE-R calibration error to about 0.02 to 0.04 across datasets, versus up to 0.89 for LaBo on Shapes3d.
• VH-CBM improves concept F1 over the best VLM-CBM by about 37 p.p. on Shapes3d, and on CUB with less than 1% annotated concepts it improves about 31.8% over CLIP zero-shot scores.

Surprising findings

• A linear probe on CLIP/DINO embeddings with the same annotations is competitive or better on calibration for Shapes3d and Derma, and on concept F1 for CelebA and Derma with DINO.
• Disentanglement benefit of VH-CBM is backbone-dependent; with CLIP on CUB it ranks lower than VLG-CBM.

Limitations

• Requires access to VLM embedding space and annotated concepts, so applicability depends on the annotation budget.
• Disentanglement gains are backbone- and dataset-dependent.
• Sequential GP training per concept takes minutes to tens of minutes per iteration on an A100.

Experimental setup

Datasets: Shapes3d CelebA CUB Derma

Benchmarks: Shapes3d CelebA CUB Derma

Metrics: F1 task (macro) F1 concept (macro) ROC-AUC DCI disentanglement ECCE-R calibration

Baselines: CBM LaBo LF-CBM VLG-CBM linear probe CLIP zero-shot

Builds on: concept bottleneck models LaBo LF-CBM VLG-CBM GP-based concept learning with Dirichlet likelihood

Closest papers by meaning

• VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.94)
• Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.92)
• V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.92)
• Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.91)
• Bayesian Concept Bottleneck Models with LLM Priors (NeurIPS 2025 · similarity 0.89)
• Learning to Intervene on Concept Bottlenecks (ICML 2024 · similarity 0.89)