Research map / Vision-language and multimodal / CLIP and zero-shot
Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALBM, a language bottleneck model whose concept space is organized as attribute-formed, class-specific concepts over a unified attribute set, plus visual attribute prompts (VAPL) and an LLM-based Description-Summary-Supplement (DSS) concept generation strategy.
Problem
Existing language bottleneck models use a class-shared concept space, which lets classifiers rely on spurious non-essential cues and prevents transfer of the concept classifier to unseen classes; manual concept collection is also costly.
Approach
Build a class-specific concept space where each class's concepts are descriptions of attributes from a cross-class unified attribute set, so each class is predicted only from its own essential concepts; novel-class classifiers are formed by similarity-weighted combination of base-class classifiers, and attribute-specific visual prompts extract fine-grained features.
Key results
- Zero-shot setting: ALBM outperforms training-free language bottlenecks on 8 of 9 datasets, with 2.0% to 20.7% gains (e.g., ImageNet 64.6 vs 44.3 for CLIP-GPT).
- Base-to-novel: ALBM improves over LaBo on 7 of 9 datasets (1.3% to 19.2%), with novel-class gains over training-free methods of 0.6% to 15.9%.
- Ablation (averaged over nine datasets): concept classifier training raises base accuracy from 54.2 to 74.0; VAPL adds 3.2 points on base classes (77.2 base, 58.2 novel).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add concept classifier training (ablation) | ALBM · Nine-dataset average, base classes | accuracy: 54.2 -> 74.0 | Helped |
| Add concept classifier training (ablation) | ALBM · Nine-dataset average, novel classes | accuracy: 55.2 -> 57.5 | Helped |
| Add Visual Attribute Prompt Learning (ablation) | ALBM · Nine-dataset average, base classes | accuracy: 74.0 -> 77.2 | Helped |
| Add Visual Attribute Prompt Learning (ablation) | ALBM · Nine-dataset average, novel classes | accuracy: 57.5 -> 58.2 | Helped |
| Use ALBM's DSS-generated attribute-formed concepts instead of the strongest existing training-free language bottleneck result | Zero-shot image recognition · Eight of nine fine-grained benchmarks | accuracy: 2.0% to 20.7% performance improvements | Helped |
| Use ALBM's attribute-formed class-specific concept space instead of existing language bottleneck models | Base-to-novel image recognition · Seven of nine benchmarks, base classes | accuracy: improvements ranging from 1.3% to 19.2% | Helped |
| Use ALBM's attribute-formed class-specific concept space instead of existing language bottleneck models | Base-to-novel image recognition · Two of nine benchmarks, base classes | accuracy: performance drops from 2.3% to 4.2% | Hurt |
| Use ALBM's attribute-formed concept space instead of existing training-free language bottlenecks | Base-to-novel image recognition · Novel classes | accuracy: 0.6% to 15.9% improvement | Helped |
Surprising findings
- Interpretable LBM methods remain well below unexplainable zero-shot CLIP on base-to-novel results, which the authors attribute to weak interpretability of CLIP visual features.
Limitations
- Performance remains far below unexplainable class-name CLIP classification.
- Aircraft zero-shot accuracy slightly lower than the best prior training-free method (18.0 vs 19.9 for CuPL)
- Relies on GPT-4o and an existing description step from LaBo's concept sets.
Experimental setup
Datasets: Aircraft CUB DTD Flowers102 Food101 OxfordPets CIFAR-10 CIFAR-100 ImageNet
Benchmarks: Aircraft CUB DTD Flowers102 Food101 OxfordPets CIFAR-10 CIFAR-100 ImageNet
Metrics: classification accuracy (zero-shot) base accuracy novel accuracy
Baselines: ZS-CLIP LP-CLIP VDCLIP CuPL CLIP-GPT LaBo CLBM
Builds on: CLIP Language Bottleneck Model (LaBo) CLIP-GPT Multimodal Attribute Prompting (MAP) Concise Language Bottleneck Model (CLBM)
Closest papers by meaning
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.88)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.88)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.87)
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.87)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.87)
- Coarse-to-Fine Concept Bottleneck Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.