Research map / Architectures and efficiency / RNNs and normalization
NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeurFlow, an automated framework that identifies core concept neurons per target neuron, clusters them into neuron groups, and builds a hierarchical circuit with IG-based interaction weights, with MLLM-based labeling and image debugging applications on ResNet50 and GoogLeNet.
Problem
Existing neuron-interpretability methods analyze individual neurons and their link to the final output, which fails when neurons are polysemantic and ignores interactions between neuron groups across layers.
Approach
Identify core concept neurons for each target neuron via integrated-gradient importance over its top-k activating image patches, cluster the core neurons into groups by shared semantic visual features, and quantify inter-group interactions as summed normalized attributions to form a hierarchical circuit per class.
Key results
- Core concept neurons yield larger loss increases than random neuron combinations in nearly all cases over 15,000 cases per layer per model.
- For tau=1, IG-based importance scores correlate above 0.6 with loss in both models and near-perfect for several ResNet50 layers.
- Overlap F1 with NeuronMCT critical neurons is 0.72 on ResNet50 and 0.79 on GoogLeNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use integrated gradients instead of other attribution methods for edge weights (ablation) | NeurFlow · ResNet50 and GoogLeNet, last 10 layers | correlation with loss | Helped |
| Use integrated gradients instead of SmoothGrad (ablation) | NeurFlow · ResNet50 and GoogLeNet, last 10 layers | correlation with loss | Helped |
| Use integrated gradients instead of Gradient Shap (ablation) | NeurFlow · ResNet50 and GoogLeNet, last 10 layers | correlation with loss | No clear effect |
| Sum edge weights rather than average them when aggregating neuron-group relation weights (ablation) | NeurFlow concept circuit · GoogLeNet and ResNet50 | correlation with loss | Mixed |
| Increase k from the baseline k=50 (ablation) | NeurFlow core concept neuron selection · ResNet50 and GoogLeNet | overlap with k=50 core neurons | No clear effect |
| Use tau=20 or tau=24 instead of smaller tau (ablation) | NeurFlow core concept neuron selection · ResNet50 and GoogLeNet | accuracy when retaining core concept neurons | Helped |
| Add 50% more non-core neurons (ablation) | NeurFlow core concept neurons · ResNet50 | accuracy when retaining neurons | No clear effect |
| Add 50% more non-core neurons (ablation) | NeurFlow core concept neurons · GoogLeNet, retaining through layer 5b | accuracy: +25% | Helped |
| Mask core concept neurons rather than random neurons (ablation) | NeurFlow · ResNet50 and GoogLeNet | accuracy | Hurt |
| Retain only core concept neurons rather than an equal number of random neurons (ablation) | NeurFlow · ResNet50 and GoogLeNet | accuracy | Helped |
| Replace NeurFlow core concept neurons with L2-weight-ranked neurons (ablation) | NeurFlow target-neuron selection · ResNet50 | loss difference: -0.082 | Hurt |
| Replace NeurFlow core concept neurons with L2-weight-ranked neurons (ablation) | NeurFlow target-neuron selection · GoogLeNet | loss difference: -0.013 | Hurt |
Surprising findings
- Masking core concept neurons at GoogLeNet layer 5a reduces accuracy to nearly 0, while masking random neurons has minimal effect.
- Knockoff attribution performs poorly at ranking neuron importance compared with other attribution methods.
Limitations
- Concept definition relies on top-k activating patches, which may miss distributed activation patterns.
- Focused on CNNs; extension to other architectures left to future work.
- Exhaustive brute-force optimality cannot be verified, only sampled random combinations.
Experimental setup
Datasets: ILSVRC2012 validation set
Benchmarks: ILSVRC2012 validation set
Metrics: loss difference vs random neuron combinations accuracy change under masking/retaining Pearson correlation of IG scores with loss F1 overlap of critical neurons logit drop for image debugging
Baselines: NeuronMCT NeuCEPT Knockoff LRP Guided Backpropagation SmoothGrad Saliency Gradient SHAP
Builds on: NeuronMCT Network Dissection integrated gradients Concept Activation Vectors NeuCEPT
Closest papers by meaning
- Evaluating Neuron Explanations: A Unified Framework with Sanity Checks (ICML 2025 · similarity 0.87)
- Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder (NeurIPS 2025 · similarity 0.85)
- Learning dynamic representations of the functional connectome in neurobiological networks (ICLR 2024 · similarity 0.85)
- Functional Connectomes of Neural Networks (AAAI 2025 · similarity 0.85)
- Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR 2024 · similarity 0.84)
- $\textit{NeuroPath}$: A Neural Pathway Transformer for Joining the Dots of Human Connectomes (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.