Research map / Vision-language and multimodal / Human and embodied modeling
Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames concept-set creation for TCAV as image generation: an RL (DQN) agent selects seed prompts and a Stable Diffusion LoRA is preference-optimized using TCAV scores as the preference signal (RLPO).
Problem
Concept-based explanations such as TCAV require humans to guess and manually collect candidate concept image sets, which is labor-intensive and tends to miss important concepts; retrieval-based automatic concept discovery stays close to the test-class pixels.
Approach
Treat concept set creation as image generation: a DQN picks seed keywords and a Stable Diffusion model is fine-tuned via LoRA with a DPO-style preference loss, where the preference between two generated image groups comes from their TCAV scores rather than human rankings.
Key results
- RL search achieved entropy 2.80, ANC 0.43, ICV 2.17 versus 2.40 entropy for the best epsilon-greedy baseline (0.25 greedy) in the search-strategy ablation.
- Human survey (n=260 laymen, n=240 experts) gave RLPO an exploration gap of 91.54% (laymen) and 65.45% (experts) versus 6.54% and 10.45% for retrieval concepts.
- RLPO concepts had lower CLIP/ResNet cosine similarity to test images (about 0.49-0.52 vs 0.69-0.77 for EAC/Lens/CRAFT) while all methods reached TCAV score 1.0; RLPO-XAIF took 56 s versus 72 s for AI feedback and 180 s for human feedback.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate concepts with RLPO rather than retrieval-based methods | GoogleNet concept explanations · Novel concepts | CS: 0.76 ± 0.03 -> 0.52 ± 0.04 | Helped |
| generate concepts with RLPO rather than retrieval-based methods | GoogleNet concept explanations · Novel concepts | ED: 7.21 ± 0.63 -> 10.48 ± 0.50 | Helped |
| generate concepts with RLPO rather than retrieval-based methods | GoogleNet concept explanations · Novel concepts | TCAV score: 1.0 | No clear effect |
| use RL search rather than 0.25 Greedy (ablation) | RLPO search on GoogleNet · Search strategy ablation | Entropy: 2.40 -> 2.80 | Helped |
| use RL search rather than 0.25 Greedy (ablation) | RLPO search on GoogleNet · Search strategy ablation | ANC: 0.21 -> 0.43 | Helped |
| use RL search rather than 0.25 Greedy (ablation) | RLPO search on GoogleNet · Search strategy ablation | ICV: 1.04 -> 2.17 | Helped |
| use RL search rather than 0.5 Greedy (ablation) | RLPO search on GoogleNet · Search strategy ablation | Entropy: 1.95 -> 2.80 | Helped |
| use RL search rather than 0.75 Greedy (ablation) | RLPO search on GoogleNet · Search strategy ablation | Entropy: 1.85 -> 2.80 | Helped |
| use TCAV feedback rather than human feedback (ablation) | RLPO · Scoring mechanisms ablation | Execution time: 180 ± 30s -> 56 ± 0.7s | Helped |
| use TCAV feedback rather than AI feedback (ablation) | RLPO · Scoring mechanisms ablation | Execution time: 72 ± 1.2s -> 56 ± 0.7s | Helped |
| replace TCAV feedback with human feedback (ablation) | RLPO · Scoring mechanisms ablation | Model-specific explanations | Hurt |
| replace TCAV feedback with AI feedback (ablation) | RLPO · Scoring mechanisms ablation | Model-specific explanations | Hurt |
Surprising findings
- Gibberish seed prompts rarely yielded meaningful concepts, showing the importance of good starting points.
- Most ML engineers in the user study missed the 'running' concept even though the generated concepts were shown to them.
Limitations
- Quality of the initial seed prompts strongly influences search trajectory and efficiency.
- Generated concepts depend on the pre-training data and biases of the generative model.
- Concepts may be non-domain-specific, so domain-specific applications (e.g., medical) may yield irrelevant images.
Experimental setup
Datasets: ImageNet CelebA IMDB sentiment
Benchmarks: ImageNet classifiers (GoogleNet, InceptionV3, ViT, Swin) CelebA Blonde classification IMDB sentiment
Metrics: TCAV score cosine similarity euclidean distance entropy average normalized count inverse coefficient of variation exploration gap odds c-deletion AUC Wasserstein distance
Baselines: EAC Lens CRAFT epsilon-greedy search RLPO-HF RLPO-AIF
Builds on: TCAV DQN Diffusion-DPO LoRA Explain Any Concept
Closest papers by meaning
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.88)
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.87)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.87)
- Concept-Centric Token Interpretation for Vector-Quantized Generative Models (ICML 2025 · similarity 0.86)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.86)
- FaCT: Faithful Concept Traces for Explaining Neural Network Decisions (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.