Research map / Vision-language and multimodal / Human and embodied modeling
A Multimodal Automated Interpretability Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MAIA, a GPT-4V-based agent that composes interpretability tools (dataset exemplars, text-to-image generation, image editing, describe/summarize) into Python experiments to describe neurons, remove spurious features, and reveal classifier biases.
Problem
Understanding neural network components requires slow, costly human experimentation; existing automated interpretability methods mostly label neurons from precomputed exemplars in a single pass and are low-precision for causal behavior.
Approach
A pretrained multimodal VLM agent is given an API of System (neuron instrumentation) and Tools (exemplar retrieval, text-to-image, image editing, description/summarization, logging) and iteratively writes and runs Python experiments with hypothesis updating to explain neuron behavior and audit models, without retraining.
Key results
- MAIA outperforms MILAN on predictive neuron description across ResNet-152, DINO, and CLIP and is often on par with human experts.
- In 2AFC on synthetic neurons, MAIA labels were preferred over MILAN (0.73) and slightly over human labels (0.53 vs human).
- Removing spurious features in ResNet-18 on Spawrious with 22 MAIA-selected neurons raised test accuracy from 0.731 to 0.837 without balanced data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 22 robust units through iterative MAIA experiments rather than MILAN-based selection | ResNet-18 classifier · Spawrious test | Test Acc.: 0.786 -> 0.837 | Helped |
| Select 22 robust units through iterative MAIA experiments rather than random selection | ResNet-18 classifier · Spawrious test | Test Acc.: 0.705 $\pm$ 0.05 -> 0.837 | Helped |
| Use MAIA-selected 22-unit subset rather than L1 Top 22 on unbalanced data | ResNet-18 classifier · Spawrious test | Test Acc.: 0.757 -> 0.837 | Helped |
| Use MAIA-selected 22-unit subset rather than all 50 selected units | ResNet-18 classifier · Spawrious test | Test Acc.: 0.779 -> 0.837 | Helped |
| Use MAIA-selected features rather than MILAN with GPT-4V | ResNet-18 classifier · Spawrious test | Test Acc.: 0.690 -> 0.837 | Helped |
| Add synthetic image generation to dataset-exemplar experiments (ablation) | MAIA neuron description · Neuron description predictive evaluation | Positive activation | Helped |
| Add dataset exemplars to generated-input experiments (ablation) | MAIA neuron description · Neuron description predictive evaluation | Positive activation | Helped |
| Replace Stable Diffusion v1.5 with DALL-E 3 for text-to-image generation (ablation) | MAIA neuron description · Neuron description predictive evaluation | Positive activation | Helped |
| Use MAIA rather than MILAN to produce synthetic-neuron descriptions | Synthetic neuron descriptions · Synthetic neurons | 2AFC preference: $0.73 \pm 4e^{-4}$ | Helped |
| Use human descriptions rather than MILAN descriptions | Synthetic neuron descriptions · Synthetic neurons | 2AFC preference: $0.83 \pm 5e^{-4}$ | Helped |
Surprising findings
- Using DALL-E 3 instead of Stable Diffusion v1.5 as the text-to-image tool improved MAIA's performance, suggesting the agent is bounded by its tools rather than its reasoning.
- MILAN labels with GPT-4V in place of its original model performed worse on the spurious feature task (0.690 vs 0.786).
Limitations
- Requires human supervision to avoid confirmation bias and drawing conclusions from small samples.
- Image generation and editing tools sometimes fail to follow instructions.
- Open-source VLM backbones currently lag far behind GPT-4V.
Experimental setup
Datasets: ImageNet validation set Spawrious CC3M MILANnotations
Benchmarks: synthetic vision neurons Spawrious ImageNet
Metrics: average activation on positive vs neutral exemplars 2AFC human agreement test accuracy
Baselines: MILAN Human experts MILAN (GPT-4V)
Builds on: Automated Interpretability Agent MILAN Stable Diffusion InstructPix2Pix Grounded DINO
Closest papers by meaning
- Measuring Per-Unit Interpretability at Scale Without Humans (NeurIPS 2024 · similarity 0.87)
- MICLIP: Learning to Interpret Representation in Vision Models (ICLR 2026 · similarity 0.87)
- Network Dissection: Quantifying Interpretability of Deep Visual Representations (CVPR 2017 · similarity 0.86)
- Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.86)
- Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment (AAAI 2025 · similarity 0.86)
- Enhancing Performance of Explainable AI Models with Constrained Concept Refinement (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.