Research map / Architectures and efficiency / RNNs and normalization
Training the Untrainable: Introducing Inductive Bias via Representational Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes guidance, which trains a target network with its task loss plus a layerwise linear-CKA dissimilarity term against a frozen guide network, so that architectural (and learned) priors transfer and "untrainable" architectures (FCNs, RNNs, plain CNNs, Transformers on parity) become trainable.
Problem
Some architectures are considered untrainable for a task (overfitting, underfitting, or poor convergence even with tuned hyperparameters), e.g., FCNs on ImageNet, vanilla RNNs on copy-paste and language modeling, Transformers on parity; the inductive biases that would fix them are poorly understood
Approach
Use a frozen guide network (trained or randomly initialized) and add a layerwise representational-alignment loss (1 - linear CKA) between mapped intermediate layers of guide and target, so the target inherits the guide's architectural prior without needing matched architectures or distillation on output logits.
Key results
- Guided RNN on copy-paste rises from 14.35% to 23.27% (Transformer guide) and 42.56% (untrained Transformer guide).
- Untrained ResNet-18 guidance raises Deep FCN top-5 accuracy from 1.65% to 13.10%.
- Guided small RNN perplexity on WikiText-103 drops from 69.19 to 40.01 (Transformer guide) vs. 34.15 for the Transformer baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Guide the RNN with a trained Transformer | RNN · Copy-Paste test | accuracy: 14.35 $\pm$ 0.01 -> 23.27 $\pm$ 1.02 | Helped |
| Use an untrained rather than trained Transformer as the RNN guide (ablation) | Guided RNN · Copy-Paste test | accuracy: 23.27 $\pm$ 1.02 -> 42.56 $\pm$ 1.51 | Helped |
| Guide the Transformer with a trained RNN | Transformer · Parity test | accuracy: 71.98 $\pm$ 3.16 -> 78.49 $\pm$ 2.16 | Helped |
| Use an untrained rather than trained RNN as the Transformer guide (ablation) | Guided Transformer · Parity test | accuracy: 78.49 $\pm$ 2.16 -> 70.38 $\pm$ 4.17 | Hurt |
| Guide the RNN with a trained Transformer | RNN · WikiText-103 Small test | perplexity: 69.19 $\pm$ 1.89 -> 40.01 $\pm$ 1.54 | Helped |
| Use an untrained rather than trained Transformer as the RNN guide (ablation) | Guided RNN · WikiText-103 Small test | perplexity: 40.01 $\pm$ 1.54 -> 59.61 $\pm$ 2.33 | Hurt |
| Guide the RNN with a trained Transformer | RNN · WikiText-103 Large test | perplexity: 89.13 $\pm$ 2.00 -> 36.91 $\pm$ 1.04 | Helped |
| Guide the Deep FCN with a trained ResNet-18 | Deep FCN · ImageNet-1K validation | Top-5 accuracy: 1.65 $\pm$ 1.21 -> 7.50 $\pm$ 1.51 | Helped |
| Use an untrained rather than trained ResNet-18 as the Deep FCN guide (ablation) | Guided Deep FCN · ImageNet-1K validation | Top-5 accuracy: 7.50 $\pm$ 1.51 -> 13.10 $\pm$ 0.72 | Helped |
| Guide the Wide FCN with a trained ResNet-18 | Wide FCN · ImageNet-1K validation | Top-5 accuracy: 34.09 $\pm$ 0.91 -> 43.01 $\pm$ 0.92 | Helped |
| Use an untrained rather than trained ResNet-18 as the Wide FCN guide (ablation) | Guided Wide FCN · ImageNet-1K validation | Top-5 accuracy: 43.01 $\pm$ 0.92 -> 39.47 $\pm$ 0.31 | Hurt |
| Guide the Deep ConvNet with a trained ResNet-50 | Deep ConvNet · ImageNet-1K validation | Top-5 accuracy: 70.02 $\pm$ 1.52 -> 78.91 $\pm$ 2.16 | Helped |
Surprising findings
- An untrained (randomly initialized) guide often outperforms a trained guide, suggesting architectural priors alone drive gains.
- Deep ConvNet without residuals benefits only from a trained ResNet-50 guide, not an untrained one.
Limitations
- Substantially higher GPU memory usage due to storing activations across layers and larger batches needed for CKA.
- Gains for FCNs remain low in absolute terms (e.g., Deep FCN 13.10% top-5), so they are not yet competitive vision models.
- Experiments are small scale (150M parameters or fewer for sequence models) and hyperparameters were not optimized beyond a learning-rate sweep.
Experimental setup
Datasets: ImageNet-1K WikiText-103 copy-paste (synthetic) parity (synthetic bitstrings)
Benchmarks: ImageNet-1K WikiText-103 copy-paste parity
Metrics: top-5 validation accuracy accuracy perplexity error consistency
Baselines: vanilla target training knowledge distillation (logit KL) ResNet-18 ResNet-50
Builds on: centered kernel alignment knowledge distillation
Closest papers by meaning
- Co-Advise: Cross Inductive Bias Distillation (CVPR 2022 · similarity 0.88)
- Identity Crisis: Memorization and Generalization Under Extreme Overparameterization (ICLR 2020 · similarity 0.87)
- Initial Guessing Bias: How Untrained Networks Favor Some Classes (ICML 2024 · similarity 0.86)
- Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness (NeurIPS 2025 · similarity 0.85)
- B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable (NeurIPS 2024 · similarity 0.85)
- PICNN: A Pathway towards Interpretable Convolutional Neural Networks (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.