Research map / Architectures and efficiency / RNNs and normalization
Layerwise Change of Knowledge in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends AND-OR interaction primitives to intermediate layers via linear probes, tracking newly emerged and forgotten interactions across layers and relating them to generalization and stability of DNNs.
Problem
How a DNN gradually extracts new knowledge and forgets noisy features across layers during forward propagation, given no consensus definition of knowledge and no alignment of intermediate-layer features.
Approach
Defines intermediate-layer knowledge by training a linear probe on each layer's features and extracting AND-OR interactions from the probe's classification confidence, so adjacent layers encode comparable interaction sets; then decomposes interaction strength into overlap, forgotten, and newly emerged terms.
Key results
- Only about 21.8 AND/OR salient interactions per MNIST image and about 45.6 per CIFAR-10 image, confirming sparsity.
- Low-order interactions show higher cross-model IoU than high-order interactions across DNNs trained on CIFAR-10 and SST-2.
- Redundancy of interactions in each layer first rises then falls through layers, indicating later layers mainly remove redundant interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use low-order rather than high-order interactions | DNN interaction representations across models trained for the same task · CIFAR-10 and SST-2 | cross-model IoU | Helped |
| Use low-order rather than high-order interactions | DNN interaction representations under Gaussian input perturbations · MNIST, CIFAR-10, and SST-2 | interaction stability | Helped |
| Vary κ from 0.04 to 0.03 or 0.05 times the output-score difference (ablation) | Interaction extraction · ResNet-20 on CIFAR-10 | interaction extraction robustness | No clear effect |
Surprising findings
- Early and middle layers encode many redundant interactions that are later forgotten, rather than building knowledge monotonically.
- Extremely high-order interactions are learned in later layers but the learning is unstable.
Limitations
- Relies on linear probes whose faithfulness to the original model's actual computation is not guaranteed.
- Interaction computation uses only a small selection of input variables to reduce cost.
- Sparsity theory for combined AND-OR interactions is not proven in general.
Experimental setup
Datasets: MNIST CIFAR-10 SST-2 UCI census commercial tabular dataset
Benchmarks: MNIST CIFAR-10 SST-2
Metrics: IoU of salient interactions completeness redundancy stability of interactions interaction strength
Baselines: Integrated Gradient Shapley value information bottleneck
Builds on: AND-OR interaction Shapley value information bottleneck probing classifiers
Closest papers by meaning
- Monitoring Primitive Interactions During the Training of DNNs (AAAI 2025 · similarity 0.90)
- Defining and extracting generalizable interaction primitives from DNNs (ICLR 2024 · similarity 0.90)
- Towards the Dynamics of a DNN Learning Symbolic Interactions (NeurIPS 2024 · similarity 0.90)
- Where We Have Arrived in Proving the Emergence of Sparse Interaction Primitives in DNNs (ICLR 2024 · similarity 0.88)
- Explaining Generalization Power of a DNN Using Interactive Concepts (AAAI 2024 · similarity 0.87)
- The Unreasonable Ineffectiveness of the Deeper Layers (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.