Research map / Architectures and efficiency / RNNs and normalization
Similarity and Matching of Neural Network Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train an affine stitching layer with task loss instead of direct least-squares matching | Same-architecture convolutional network pairs with different initializations · ResNet-20 and Tiny-10, CIFAR-10 | relative accuracy | Helped |
| Initialize task-loss stitching from the least-squares solution rather than random initialization (ablation) | Tiny-10 stitching · CIFAR-10 | relative accuracy | Helped |
| Increase ResNet-20 width | Task-loss stitching · ResNet-20, CIFAR-10 | relative accuracy | Helped |
| Use task-loss matching rather than activation-space SVD least-squares matching at a fixed rank | Low-rank Tiny-10 stitching · Tiny-10, CIFAR-10 | relative accuracy | Helped |
| Penalize CKA while retaining the task cross-entropy objective (ablation) | A model stitched with itself · ResNet, Layer 2.0 | accuracy: lower than 0.7 | No clear effect |
| Increase sparsity using L1 regularization (ablation) | Direct least-squares matching · Tiny-10, CIFAR-10 | relative accuracy: 80-90 % | Hurt |
| Increase sparsity using L1 regularization (ablation) | Task-loss matching · Tiny-10, CIFAR-10 | relative accuracy | Helped |
| Weight direct matching toward high activation values (ablation) | Weighted mean-squared direct matching · Tiny-10 | relative accuracy | Helped |
| Weight direct matching by gradients of the network output (ablation) | Weighted mean-squared direct matching · Tiny-10 | relative accuracy | No clear effect |
| Train stitching from random rather than least-squares initialization (ablation) | Task-loss matching · Tiny-10 | relative accuracy | Mixed |
| Use task loss rather than direct activation matching (ablation) | Sparse stitching · Tiny-10, CIFAR-10 | relative accuracy | Helped |
| Stitch networks trained on different datasets | ImageNet-to-CelebA stitching · CelebA | relative accuracy: 3% relative drop | Hurt |
Closest papers by meaning
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.92)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.88)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.88)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.87)
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.87)
- Mechanistic Permutability: Match Features Across Layers (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.