Research map / Label-efficient and robust learning / Recognition and long-tail
Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOT-CBM, a concept bottleneck model that aligns ViT patch features with textual concept embeddings via a disentangled optimal transport assignment, with saliency and co-occurrence priors, to give fine-grained inversion masks and better accuracy and OOD robustness.
Problem
Existing concept bottleneck models learn only coarse image-level visual-concept relations, producing spurious, mislocalized concept predictions and offering no explicit region-level explanations, and they are hurt by data biases such as background and concept co-occurrence shortcuts.
Approach
Model patch-to-concept prediction as an entropic optimal transport problem between ViT patch embeddings and concept embeddings, so the transport plan gives both concept activations and inversion masks; add orthogonal projection losses for disentanglement and saliency-based and co-occurrence-based transport priors to counter shortcuts.
Key results
- DOT-CBM reaches 83.84% on ImageNet, 85.39% on CUB, and 85.83% on CIFAR100, improving over the best baseline by 0.91, 3.29 and 1.08 points.
- On part detection, DOT-CBM gets mAP0.5 of 50.12 on PartImageNet and 53.47 on CUB versus 41.17 and 45.16 for CoopCBM.
- On OOD CUB with background shift, DOT-CBM reaches 49.7 OOD accuracy versus 39.4 for LaBo, a gain of about 10 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DOT-CBM instead of the strongest listed baseline | image classification models · ImageNet-1K | classification accuracy: 82.93 -> 83.84 | Helped |
| use DOT-CBM instead of CoopCBM | image classification models · CUB | classification accuracy: 82.10 -> 85.39 | Helped |
| use DOT-CBM instead of the strongest listed baseline | image classification models · CIFAR-100 | classification accuracy: 84.75 -> 85.83 | Helped |
| use DOT-CBM instead of CoopCBM | image classification models · AwA2 | classification accuracy: 97.08 -> 96.83 | Hurt |
| use DOT-CBM instead of CoopCBM | part detection models · PartImageNet | mAP0.5: 41.17 -> 50.12 | Helped |
| use DOT-CBM instead of CoopCBM | part detection models · CUB | part detection mAP0.5: 45.16 -> 53.47 | Helped |
| use DOT-CBM with prior instead of CoopCBM | OOD generalization models · CUB OOD | accuracy: 36.2 -> 49.7 | Helped |
| add patch-set and concept-set priors | DOT-CBM · CUB OOD | accuracy: 36.9 -> 49.7 | Helped |
| add patch-set and concept-set priors | DOT-CBM · Dogs OOD | accuracy: 40.8 -> 52.4 | Helped |
| add OT assignment to the baseline (ablation) | DOT-CBM baseline · CUB | classification accuracy: 80.92 -> 81.75 | Helped |
| add OT assignment to the baseline (ablation) | DOT-CBM baseline · CUB | part detection mIOU: 0.41 -> 0.49 | Helped |
| add patch-set and concept-set priors to OT (ablation) | DOT-CBM · CUB | classification accuracy: 81.75 -> 84.92 | Helped |
Surprising findings
- The model learns a highly sparse concept-to-class weight matrix without explicit L1 regularization, attributed to the orthogonal losses.
Limitations
- Baselines were re-implemented with CDL concepts and a ViT backbone, so comparisons depend on this setup.
- AwA2 accuracy is slightly below CoopCBM (96.83 vs 97.08).
Experimental setup
Datasets: CUB AwA2 ImageNet-1K CIFAR-100 Places365 Dogs PartImageNet PASCAL-Part RIVAL-10
Benchmarks: ImageNet-1K CUB CIFAR100 AwA2 PartImageNet RIVAL-10 PASCAL-Parts CUB OOD (color background shift) Dogs OOD
Metrics: classification accuracy mAP0.5 mIOU
Baselines: Vanilla-CBM CEM LaBo SparseCBM CoopCBM
Builds on: Concept Bottleneck Models Sinkhorn optimal transport LaBo CDL concept generation
Closest papers by meaning
- Coarse-to-Fine Concept Bottleneck Models (NeurIPS 2024 · similarity 0.88)
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.87)
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.87)
- Show and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Models (CVPR 2025 · similarity 0.87)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.87)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.