Research map / Vision-language and multimodal / Human and embodied modeling
LG-CAV: Train Any Concept Activation Vector with Language Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LG-CAV, which trains concept activation vectors without labeled images by mimicking CLIP-computed concept activations on probe images, plus Gaussian alignment, concept ensemble, and deviation sample reweighting, and uses the resulting CAVs for model correction via activation sample reweighting.
Problem
CAV training needs many high-quality labeled concept images, which are costly to curate and limit concepts to a predefined set.
Approach
Train a CAV on a common pool of probe images so its cosine activations mimic CLIP text-concept activations on those images, after Gaussian alignment of the two activation distributions; then reuse the high-quality LG-CAVs to reweight training samples for model correction.
Key results
- On Broden across nine ImageNet backbones, the full LG-CAV (GA+CE+DSR) reaches 77.45% concept accuracy on ResNet-18 vs 68.92% for the original CAV.
- Concept-to-class accuracy on ResNet-18 rises from 6.20% (original CAV) to 24.58% with the full LG-CAV.
- Model correction on ImageNet-40 improves accuracy by up to 0.91 points over the original model, and on whole ImageNet by up to 0.68 points, with gains reported on CUB-200-2011 and CIFAR-100 as well.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LG-CAV + GA + CE + DSR instead of OA-TCAV | CAV training · Broden, ResNet-18 | concept accuracy (%): 72.62 -> 77.45 | Helped |
| Use LG-CAV + GA + CE + DSR instead of OA-TCAV | CAV training · Broden, ResNet-18 | concept-to-class accuracy: 10.11 -> 24.58 | Helped |
| Apply LG-CAV-based activation sample reweighting | ImageNet-40 model correction, ResNet-18 · ImageNet-40 | accuracy (%): 90.65 -> 91.16 | Helped |
| Apply LG-CAV-based activation sample reweighting | ImageNet model correction, ResNet-18 · ImageNet | accuracy (%): 69.76 -> 70.26 | Helped |
| Add Gaussian alignment (ablation) | LG-CAV · Broden, ResNet-18 | concept accuracy (%): 67.23 -> 74.89 | Helped |
| Add Gaussian alignment (ablation) | LG-CAV · Broden, ResNet-18 | concept-to-class accuracy: 4.72 -> 16.72 | Helped |
| Add concept ensemble (ablation) | LG-CAV + GA · Broden, ResNet-18 | concept accuracy (%): 74.89 -> 76.41 | Helped |
| Add concept ensemble (ablation) | LG-CAV + GA · Broden, ResNet-18 | concept-to-class accuracy: 16.72 -> 19.14 | Helped |
| Add deviation sample reweighting (ablation) | LG-CAV + GA + CE · Broden | concept accuracy | Helped |
| Select the most and least VL-activated images instead of random probe images (ablation) | LG-CAV training · Broden | CAV quality | Helped |
| Increase probe-image count while it is below saturation (ablation) | LG-CAV training · Broden | CAV quality | Helped |
| Increase probe-image count beyond saturation (ablation) | LG-CAV training · Broden | CAV quality | No clear effect |
Surprising findings
- The base LG-CAV without Gaussian alignment has lower concept accuracy than the original CAV (63-67% vs 65-73%), showing naive VL-guided mimicry hurts quality.
Limitations
- Intermediate-layer LG-CAVs have much lower quality and high training cost due to large dimensionality.
- Model correction gains are small (under 1 point on ImageNet).
Experimental setup
Datasets: Broden ImageNet CUB-200-2011 CIFAR-100
Benchmarks: Broden ImageNet ImageNet-40 CUB-200-2011 CIFAR-100
Metrics: concept accuracy concept-to-class accuracy TCAV score Recall@100 top-1 accuracy
Baselines: Original CAV Text-to-Concept OA-TCAV Concept Gradient Concept_Distillation Knowledge Distillation Label-free CBM HiBug PCBM Trustworthy CBM
Builds on: Concept Activation Vector (TCAV) CLIP Text-to-Concept OA-TCAV Concept_Distillation
Closest papers by meaning
- FastCAV: Efficient Computation of Concept Activation Vectors for Explaining Deep Neural Networks (ICML 2025 · similarity 0.86)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.86)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.86)
- TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.86)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.85)
- Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.