Research map / Vision-language and multimodal / CLIP and zero-shot
Explaining Domain Shifts in Language: Concept Erasing for Interpretable Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LanCE, a plug-in domain descriptor orthogonality (DDO) regularizer that uses LLM-generated domain descriptors and CLIP text embeddings to erase domain-specific concepts from concept bottleneck models for better OOD generalization.
Problem
Concept-based models built on VLMs degrade under domain shifts because domain-specific concepts (e.g., color, texture) become biased predictors, hurting out-of-distribution generalization.
Approach
Use CLIP to show visual domain shifts can be expressed in language via domain descriptors from an LLM; simulate domain-specific concept activations as concept embeddings projected onto text-embedding domain shift vectors, and add a DDO loss forcing the final classifier weights to be orthogonal to these activations.
Key results
- DDO improves OOD accuracy of CBMs by about 3 points on average across single-unseen-domain benchmarks while preserving ID accuracy.
- With LaBO on CUB-Painting, DDO raises OOD accuracy from 56.24 to 59.60 and on AwA2-clipart from 84.15 to 87.66.
- On LADV-3D, LaBO+DDO reaches 68.01 OOD vs 63.17 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DDO to LaBO | LaBO · CUB-Painting OOD | accuracy: 56.24 -> 59.60 | Helped |
| Add DDO to LaBO | LaBO · AwA2-clipart OOD | accuracy: 84.15 -> 87.66 | Helped |
| Add DDO to LaBO | LaBO · LADA-Sculpture OOD | accuracy: 74.56 -> 80.00 | Helped |
| Add DDO to LaBO | LaBO · LADV-3D OOD | accuracy: 63.17 -> 68.01 | Helped |
| Use only domain-irrelevant descriptors in DDO (ablation) | LaBO · CUB-Painting OOD | accuracy: 56.24 -> 57.60 | Helped |
| Use all domain descriptors in DDO instead of only domain-irrelevant descriptors (ablation) | LaBO · CUB-Painting OOD | accuracy: 57.60 -> 59.60 | Helped |
| Use only domain-irrelevant descriptors in DDO (ablation) | LaBO · AwA2-clipart OOD | accuracy: 84.15 -> 85.50 | Helped |
| Use all domain descriptors in DDO instead of only domain-irrelevant descriptors (ablation) | LaBO · AwA2-clipart OOD | accuracy: 85.50 -> 87.66 | Helped |
| Use only domain-irrelevant descriptors in DDO (ablation) | LaBO · LADA-Sculpture OOD | accuracy: 74.56 -> 77.70 | Helped |
| Use all domain descriptors in DDO instead of only domain-irrelevant descriptors (ablation) | LaBO · LADA-Sculpture OOD | accuracy: 77.70 -> 80.00 | Helped |
| Use only domain-irrelevant descriptors in DDO (ablation) | LaBO · LADV-3D OOD | accuracy: 63.17 -> 65.46 | Helped |
| Use all domain descriptors in DDO instead of only domain-irrelevant descriptors (ablation) | LaBO · LADV-3D OOD | accuracy: 65.46 -> 68.01 | Helped |
Surprising findings
- Using only LLM-generated domain-shared concepts helps only in certain source-domain cases, while DDO is more consistent across sources.
Limitations
- Generalization from 2D images to sculpture and 3D model images remains limited for all models.
- Performance depends on the number and relevance of LLM-generated domain descriptors.
- Domain descriptor sets are generated once and may not cover all unseen domains.
Experimental setup
Datasets: CUB-Painting PACS OfficeHome DomainNet AwA2 LAD-animal LAD-vehicle
Benchmarks: CUB-Painting AwA2-clipart LADA-Sculpture LADV-3D PACS OfficeHome DomainNet
Metrics: accuracy (ID) accuracy (OOD)
Baselines: CLIP-ZS CLIP-LP CLIP-CBM PCBM LaBO
Builds on: Concept bottleneck models CLIP-CBM PCBM LaBO CLIP
Closest papers by meaning
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.89)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.88)
- Approximate Domain Unlearning for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.87)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.87)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.