Research map / Vision-language and multimodal / Human and embodied modeling
Language Models Can Explain Visual Features via Steering
Javier Ferrando, Enrique Lopez-Cuena, Pablo Agustin Martin-Torres, Daniel Hinjos, Anna Arias-Duart, Dario Garcia-Gasulla
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition Top-k explainer on SAE-steering intervention (Steering-informed Top-k) | Top-k explanations, Gemma 3 27B · Middle-layer SAE, ImageNet test (heatmaps) | Synthetic activation score: 0.364 -> 0.505 | Improved |
| condition Top-k explainer on SAE-steering intervention | Top-k explanations, Gemma 3 27B · Middle-layer SAE, ImageNet test (heatmaps) | AUROC: 0.791 -> 0.838 | Improved |
| condition Top-k explainer on SAE-steering intervention | Top-k explanations, InternVL3-14B · Middle-layer SAE, ImageNet test (masks) | Synthetic activation score: 0.187 -> 0.254 | Improved |
| condition Top-k explainer on SAE-steering intervention | Top-k explanations, Gemma 3 27B · Middle-layer SAE, ImageNet test (masks) | CLIP score: 0.190 -> 0.193 | No clear effect |
| steering-only explanation instead of Top-k | Gemma 3 27B · Middle-layer SAE, ImageNet test | AUROC: 0.723 -> 0.675 (masks) | Hurt |
| steering-only explanation instead of Top-k | InternVL3-14B · Middle-layer SAE, ImageNet test | IoU score: 0.224 -> 0.220 (masks) | No clear effect |
| scale explainer LM from 4B to 12B to 27B (ablation) | Steering, Gemma 3 · Gemma 3 vision encoder SAE | Mean synthetic activation score and AUROC | Improved |
| filter Top-k explanations by similarity to Steering explanations (ablation) | Top-k, Gemma 3 and InternVL3 · Middle-layer SAE | Normalized synthetic activation score and AUROC | Improved |
| filter Top-k by Steering similarity (ablation) | Top-k, Gemma 3 · Middle-layer SAE | CLIP score | No clear effect |
| use steering instead of top-activating images | Explanation of background features, Gemma 3 · 300 features with largest IoU gap (heatmaps) | % explanations categorized animal: 47.7% -> 0.0% | Improved |
| add steering to Top-k | Top-k, Gemma 3 · 300 features with largest IoU gap (heatmaps) | % animal explanations (contextual bias): 47.7% -> 38.6% | Improved |
| increase evaluation set size for Top-k (ablation) | Top-k explanations · ImageNet test, 1k to 50k images | Synthetic scores, IoU, CLIP | Mixed |
Closest papers by meaning
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.91)
- Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering (NeurIPS 2026 · similarity 0.90)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.89)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
