Research map / Architectures and efficiency / RNNs and normalization
How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that task loss matching (affine stitching trained on task loss) yields misleading layer similarities because it creates out-of-distribution representations, and proposes direct matching (a least-squares affine map to the target representation) as a better similarity index.
Problem
Model stitching with task loss matching is used to measure functional similarity of neural network representations, but it can declare distant or non-corresponding layers highly similar, even a layer more similar to other layers than to itself.
Approach
Attributes TLM's failures to OOD stitched representations that fool the receiver, using energy-based OOD detectors per layer; proposes direct matching (closed-form affine mapping minimizing Frobenius distance to target representation) as a hybrid structural-functional similarity index that stays in-distribution and passes sanity checks.
Key results
- Task loss matching identifies the same layer within a ResNet-18 only 63.75% of the time (ViT-Ti 24.17%), while direct matching (functional and structural) reaches 100% intra-network.
- Inter-network layer identification on ResNet-18: LCKA 96.11%, structural direct matching 92.50%, functional direct matching 66.39%, task loss matching 35.28%.
- In the sensitivity test, functional direct matching has the highest Spearman rank correlation on CIFAR-10 ResNet (0.9169) and is among the top measures across settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use direct matching instead of task loss matching for stitching | ResNet-18 and ViT-Ti layer identification · CIFAR-10 intra-network | accuracy: 63.75% -> 100%; 24.17% -> 100% | Helped |
| Use direct matching instead of task loss matching for stitching | ResNet-18 and ViT-Ti layer identification · CIFAR-10 inter-network | accuracy: 35.28% -> 66.39%; 10.00% -> 11.85% | Mixed |
| Use functional direct matching rather than PWCCA | ResNet-18 similarity index · CIFAR-10 sensitivity test | Kendall's τ: 0.6276 -> 0.7896 | Helped |
| Use functional direct matching rather than PWCCA | ViT-Ti similarity index · CIFAR-10 sensitivity test | Kendall's τ: 0.8260 -> 0.7448 | Hurt |
| Use functional direct matching rather than LCKA | ResNet-18 similarity index · SVHN specificity test | Kendall's τ: 0.9288 -> 0.8895 | Hurt |
| Use functional direct matching rather than LCKA | ViT-Ti similarity index · ImageNet specificity test | Kendall's τ: 0.7141 -> 0.7912 | Helped |
| Train the stitcher with task loss matching rather than direct matching | ResNet-18 stitching · CIFAR-10 intra-network layer identification | accuracy: 100% -> 63.75% | Hurt |
| Train the stitcher with task loss matching rather than direct matching | ViT-Ti stitching · CIFAR-10 intra-network layer identification | accuracy: 100% -> 24.17% | Hurt |
| Optimize the stitching transformation for task loss rather than representation distance | Model stitching similarity · CIFAR-10 | OOD AUROC | Hurt |
Surprising findings
- Task loss matching can indicate that very distant layers are more similar to a layer than the layer itself, even though its initialization is the identity-like direct matching solution.
- Ding et al.'s recommended OPD shows inconsistent results across tests, questioning its use as a preferred functional baseline.
Limitations
- Experiments limited to image classifiers (ResNet-18, ViT-Ti) on CIFAR-10, SVHN, ImageNet; generality to other domains unverified.
- Evaluation is very computationally expensive (about 1.5 GPU-years), limiting scale of study.
- Results depend on the chosen OOD detector and auxiliary dataset; direct matching's ID preference is an empirical observation.
Experimental setup
Datasets: CIFAR-10 SVHN ImageNet-1k CIFAR-5M 300K Random Images
Benchmarks: CIFAR-10 SVHN ImageNet-1k
Metrics: relative accuracy AUROC accuracy of identifying corresponding layers Kendall's tau Spearman's rho
Baselines: task loss matching LCKA PWCCA OPD DM (struct.)
Builds on: model stitching energy-based OOD detection centered kernel alignment projection-weighted CCA Ding et al. statistical tests for functional similarity
Closest papers by meaning
- Similarity and Matching of Neural Network Representations (NeurIPS 2021 · similarity 0.92)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.88)
- Functional Alignment Can Mislead: Examining Model Stitching (ICML 2025 · similarity 0.87)
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.87)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.86)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.