Research map / Vision-language and multimodal / Human and embodied modeling
V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a concept bottleneck model without LLMs by quantizing unlabeled web images into a filtered common-word concept vocabulary using a CLIP ViT-L/14 Vision-to-Concept tokenizer, then training only a class-concept weight matrix.
Problem
Concept bottleneck models need costly expert-annotated concepts, and LLM-generated concepts are often verbose and non-visual, which hurts accuracy and interpretability.
Approach
Use common words (with adjective-noun bigrams and relation trigrams) as the concept vocabulary, filter them with CLIP similarity on unlabeled images selected per class, and quantize each image to its nearest concept embeddings (V2C tokenizer) to form a vision-oriented concept bottleneck coupled with the VLM, with no LLM and no concept training.
Key results
- On the ten-dataset comparison, V2C-CBM reaches 84.1% on ImageNet, above the CLIP ViT-L/14 linear probe (83.9%) and LaBo (84.0%), and is best or tied on most datasets (e.g., CUB 83.0%, DTD 78.2%).
- Averaged over datasets, V2C-CBM trails LaBo at 1-shot (57.8 vs 63.0) and 2-shot (64.0 vs 67.7) but matches or exceeds it from 4-shot onward (e.g., 79.7 vs 79.3 at 16-shot, 85.6 vs 85.2 at all shots).
- Replacing the base vocabulary with one lacking food-related words drops Food 1-shot accuracy by up to 28.9 points, showing the concept filtering selects task-relevant concepts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V2C-CBM instead of LaBo | CLIP ViT-L/14 CBM · ImageNet | accuracy: 84.0{ $\pm$0.06} -> \textbf{84.1}{ $\pm$0.05} | Helped |
| Use V2C-CBM instead of the strongest reported baseline | CLIP ViT-L/14 CBM · CIFAR100 | accuracy: \textbf{86.4} vs. 86.4{ $\pm$0.01} | No clear effect |
| Use V2C-CBM instead of linear probing | CLIP ViT-L/14 CBM · Aircraft, CUB, DTD, Flower, Food-101, HAM10000, RESISC45 | accuracy | Hurt |
| Use V2C-CBM instead of LaBo | CLIP ViT-L/14 CBM · All datasets, 1-shot | accuracy: 63.0 -> 57.8 | Hurt |
| Use V2C-CBM instead of LaBo | CLIP ViT-L/14 CBM · All datasets, 8-shot | accuracy: 75.1 -> 75.8 | Helped |
| Increase unlabeled image set from 1k to 200k (ablation) | V2C-CBM · CIFAR10, CUB, DTD, RESISC45 | accuracy: CIFAR10: 97.6 -> 98.0; CUB: 80.3 -> 83.0; DTD: 73.1 -> 78.0; RESISC45: 90.2 -> 92.6 | Helped |
| Add bigram and trigram concepts to atomic vocabulary (ablation) | V2C-CBM · Food and RESISC45, 1-shot | accuracy: Food: 52.3 -> 58.6; RESISC45: 60.5 -> 62.0 | Helped |
| Use class-name text features instead of few-shot image features (ablation) | V2C tokenizer · Flower-102, 1/2/4/8/16-shot | accuracy: 77.2 -> 88.4; 86.6 -> 93.0; 93.8 -> 96.5; 96.4 -> 97.2; 97.7 -> 98.2 | Helped |
| Use task-unrelated vocabulary instead of the base vocabulary (ablation) | V2C-CBM · Food, 1-shot | accuracy: 58.6 -> 29.7 | Hurt |
| Restrict vocabulary to food-descriptive words (ablation) | V2C-CBM · Food, 1-shot | accuracy: 58.6 -> 50.2 | Hurt |
| Initialize weights with concept priors rather than random values (ablation) | V2C-CBM · All datasets, 1/2-shot vs. 4/8/16-shot/all | accuracy: 1-shot: 50.4 -> 57.8; 2-shot: 61.6 -> 64.0; 4-shot: 71.1 -> 70.3; 8-shot: 75.8 -> 74.7; 16-shot: 79.7 -> 78.7; all: 85.7 | Mixed |
| Vary vocabulary size from 3k to 30k (ablation) | V2C-CBM · RESISC45 | accuracy: 3k: 92.5; 5k: 92.7; 10k: 92.6; 15k: 92.5; 20k: 92.5; 30k: 92.4 | No clear effect |
Surprising findings
- Class names as base features outperform few-shot image features for concept generation, especially at low shots (88.4 vs 77.2 at 1-shot on Flower).
Limitations
- Underperforms LLM-guided LaBo in 1- and 2-shot settings, especially on fine-grained datasets such as Aircraft and Food.
- Concepts may be missed when object parts are occluded in the few shot images, so concept quality depends on the labeled images.
- The concept vocabulary is limited to common words, so it cannot express open-vocabulary attributes.
Experimental setup
Datasets: ImageNet CIFAR10 CIFAR100 Aircraft CUB DTD Flower Food-101 RESISC45 HAM10000
Benchmarks: CIFAR10 CIFAR100 ImageNet Aircraft CUB DTD Flower Food-101 RESISC45 HAM10000
Metrics: classification accuracy
Baselines: LaBo CDM DCLIP DN-CBM Linear probe
Builds on: concept bottleneck models LaBo CLIP DN-CBM CDM
Closest papers by meaning
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.93)
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.92)
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.91)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.89)
- Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability (CVPR 2025 · similarity 0.88)
- Object-Centric Concept-Bottlenecks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.