Research map / Vision-language and multimodal / CLIP and zero-shot
Interpreting the Second-Order Effects of Neurons in CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interprets CLIP MLP neurons via a "second-order lens" that traces each neuron's effect through later attention heads, approximates it as one text-image direction, decomposes it into sparse text descriptions, and applies this to adversarial image generation and zero-shot segmentation.
Problem
Individual neurons in CLIP are hard to interpret: their direct (first-order) effects are negligible, indirect effects are masked by self-repair, and there are too many neurons for manual analysis.
Approach
Model each neuron's contribution to the output as a second-order effect flowing through all later attention OV circuits to the projection; this effect is sparse across images and approximately rank-1, so it can be decomposed via sparse coding into text representations in CLIP's joint space.
Key results
- Zero-shot segmentation on ImageNet-Segmentation reaches 78.1 pixel accuracy, 59.0 mIoU and 84.9 mAP, beating TextSpan (76.5 / 58.1 / 84.1).
- Second-order adversarial generation fools the binary classifier on up to 22.7 of 100 images (dog to deer), versus at most 6.3 for the indirect-effect baseline.
- Each neuron's second-order effect is significant for under 2% of images; a single principal component reconstructs it with negligible accuracy drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use second-order neuron decompositions instead of indirect-effect decompositions to select adversarial cues (ablation) | CLIP ViT-B-32 adversarial image generation · CIFAR-10 dog → deer | adversarial success count (out of 100): 6.3 -> 22.7 | Helped |
| Use second-order neuron decompositions instead of similar-word cues | CLIP ViT-B-32 adversarial image generation · CIFAR-10 bird → frog | adversarial success count (out of 100): 5.0 -> 8.0 | Helped |
| Use second-order neuron decompositions instead of random neurons | CLIP ViT-B-32 adversarial image generation · CIFAR-10 ship → truck | adversarial success count (out of 100): 0.0 -> 5.7 | Helped |
| Use second-order neuron decompositions instead of random neurons | CLIP ViT-B-32 adversarial image generation · CIFAR-10 ship → automobile | adversarial success count (out of 100): 1.3 -> 7.0 | Helped |
| Use second-order neuron decompositions instead of indirect-effect decompositions to select adversarial cues (ablation) | CLIP ViT-B-32 adversarial image generation · CIFAR-10 horse → automobile | adversarial success count (out of 100): 2.8 -> 5.3 | Helped |
| Use second-order neuron decompositions instead of TextSpan | CLIP ViT-B-32 zero-shot segmentation · ImageNet-Segmentation | Pixel accuracy: 76.5 -> 78.1 | Helped |
| Use second-order neuron decompositions instead of TextSpan | CLIP ViT-B-32 zero-shot segmentation · ImageNet-Segmentation | mIoU: 58.1 -> 59.0 | Helped |
| Use second-order neuron decompositions instead of TextSpan | CLIP ViT-B-32 zero-shot segmentation · ImageNet-Segmentation | mAP: 84.1 -> 84.9 | Helped |
| Mean-ablate second-order effects only on the 100 images with the largest effect norms (ablation) | CLIP ViT-B-32 second-order lens · ImageNet validation | zero-shot classification accuracy | Hurt |
| Replace full second-order effects with their first-principal-component approximation (ablation) | CLIP ViT-B-32 second-order lens · ImageNet validation | zero-shot classification accuracy | No clear effect |
| Use indirect effects instead of second-order effects (ablation) | CLIP ViT-B-32 neuron analysis · ImageNet validation | accuracy drop after mean-ablation | Hurt |
| Increase the sparse text description set size to 128 per neuron (ablation) | CLIP ViT-B-32 sparse neuron decomposition · ImageNet validation (10%) | zero-shot classification accuracy | Helped |
Surprising findings
- Indirect (mean-ablation) effects are obscured by self-repair, so ablating a neuron yields smaller accuracy drops than its second-order effect suggests.
- For the ship-to-truck task, baselines generated no adversarial images while the second-order method generated 5.7 on average.
Limitations
- Ignores neuron effects on attention queries and keys, analyzing only values.
- Does not model neuron-neuron interactions within or across layers.
- Adversarial pipeline relies on several independent components (LLM, text-to-image model) that can fail; manual filtering of generated images.
Experimental setup
Datasets: ImageNet ImageNet-R CIFAR-10 ImageNet-Segmentation LAION
Benchmarks: ImageNet ImageNet-R ImageNet-Segmentation CIFAR-10
Metrics: zero-shot accuracy pixel accuracy mIoU mAP adversarial success count
Baselines: Partial-LRP Rollout LRP GradCAM TextSpan indirect effect similar words
Builds on: logit lens TextSpan orthogonal matching pursuit mean-ablation
Closest papers by meaning
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.86)
- Discovering Influential Neuron Path in Vision Transformers (ICLR 2025 · similarity 0.85)
- Boosting the visual interpretability of CLIP via adversarial fine-tuning (ICLR 2025 · similarity 0.85)
- CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding (ICLR 2024 · similarity 0.84)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.84)
- LINE: LLM-based Iterative Neuron Explanations for Vision Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.