Research map / Label-efficient and robust learning / Recognition and long-tail
Functional Alignment Can Mislead: Examining Model Stitching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that model stitching can align networks trained on very different information (shortcut colour vs shape, birdsong vs ImageNet, clustered noise, autoencoders on different datasets), arguing functional alignment does not imply informational similarity.
Problem
Model stitching is used as a functional similarity measure under the assumption that stitch-compatible representations capture similar information; the paper questions whether this holds, especially for models that learn shortcuts.
Approach
Controlled shortcut-learning setups (coloured MNIST variants) and real-world pairs (ImageNet vs Stylized ImageNet, ImageNet vs birdsong, clustered random noise, autoencoders on different datasets) show that stitching layers can connect representations that encode different information, so stitching accuracy is not evidence of informational similarity.
Key results
- Models trained on different biases (e.g. Colour vs Digit) stitch into the Digit receiver with accuracy above the receiver's own baseline at all layers.
- Clustered-Noise representations stitch into an ImageNet ResNet-50 with 100% top-1 accuracy at Res Block 3 onward, versus a 76.13% baseline.
- ImageNet to birdsong stitching reaches 88.40% top-1 at the Linear layer against a 77.98% birdsong baseline, despite different tasks and modalities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Stitch an ImageNet sender into the Stylized receiver before the classifier | ResNet-50 · Stylized ImageNet | top-1 accuracy: 60.18 -> 74.34 | Helped |
| Stitch an ImageNet sender into the Stylized receiver before the classifier | ResNet-50 · Stylized ImageNet | top-5 accuracy: 82.60 -> 91.93 | Helped |
| Stitch a Stylized sender into the ImageNet receiver before the classifier | ResNet-50 · ImageNet | top-1 accuracy: 76.13 -> 49.17 | Hurt |
| Stitch a Stylized sender into the ImageNet receiver before the classifier | ResNet-50 · ImageNet | top-5 accuracy: 92.86 -> 72.30 | Hurt |
| Stitch a 10-class ImageNet sender into the Birdsong receiver before the classifier | ResNet-50 · Birdsong recognition (10-class xeno-canto) | top-1 accuracy: 77.98 -> 88.40 | Helped |
| Stitch a 10-class ImageNet sender into the Birdsong receiver before the classifier | ResNet-50 · Birdsong recognition (10-class xeno-canto) | top-5 accuracy: 96.07 -> 99.80 | Helped |
| Stitch clustered random noise into the ImageNet receiver before Block 4 | ResNet-50 · ImageNet | top-1 accuracy: 76.13 -> 100 | Helped |
| Stitch clustered random noise into the ImageNet receiver before Block 4 | ResNet-50 · ImageNet | top-5 accuracy: 92.86 -> 100 | Helped |
| Stitch Colour-Only representations into a Greyscale MNIST receiver | MNIST-trained model · Greyscale MNIST (Colour-Only stitch data) | accuracy: 99.0 ± 0.1% -> 100% | Helped |
| Use Colour-Only instead of Correlated or Digit sender data | Digit receiver · Correlated MNIST | test accuracy | Helped |
| Use clustered random noise as the sender representation instead of learned task representations | Digit receiver · Correlated MNIST | test accuracy | Helped |
| Remap class labels with an offset of 1 while training the stitch | Greyscale MNIST, Colour-Only, and Correlated models · MNIST-derived classification tasks | test accuracy | Mixed |
Surprising findings
- Stitching from a Digit sender into Colour or Correlated receivers also exceeds baseline at some layers, contradicting the idea that stitching accuracy ranks representation quality.
- Clustered random noise can be stitched into a trained ImageNet model with full accuracy at later layers.
Limitations
- Stitching failures (e.g. Stylized into ImageNet) are hard to interpret, since lack of stitch compatibility may reflect a poor linear mapping or task difficulty.
- Only stitching between identical architectures at matching points and with a restricted 1x1 conv stitch is studied.
- Many experiments use small, simple controlled datasets, and the authors acknowledge results may not generalise to harder settings.
Experimental setup
Datasets: Colour MNIST (Correlated, Digit, Colour, Colour-Only) Greyscale MNIST ImageNet Stylized ImageNet xeno-canto birdsong (10-class) Fashion-MNIST CIFAR-10 Clustered-Noise (synthetic)
Benchmarks: Correlated colour MNIST MNIST ImageNet Stylized ImageNet xeno-canto birdsong
Metrics: test accuracy top-1 accuracy top-5 accuracy numerical rank of feature maps
Baselines: CKA Stitching with receiver baseline (Bansal et al., 2021) Moschella et al. (2022) relative representations stitching
Builds on: model stitching (Lenc & Vedaldi, 2015) Anna Karenina Hypothesis (Bansal et al., 2021) Platonic Representation Hypothesis (Huh et al., 2024) Colour MNIST (Bahng et al., 2020)
Closest papers by meaning
- Towards a learning theory of representation alignment (ICLR 2025 · similarity 0.91)
- Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.90)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.87)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.87)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.87)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.