Research map / Architectures and efficiency / Neural architecture search

Network of Theseus (Like the ship)

NeurIPS 2026 · Architectures and efficiency › Neural architecture search · approach: Latency-aware architecture search · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Network of Theseus, a progressive part-by-part conversion that replaces components of a (trained or untrained) guide network with target modules aligned via CKA or D-MNN, then fine-tunes the converted target on ImageNet or Wikitext-103.

Problem

Standard deep learning assumes the architecture used for training must be the one deployed at inference, which limits architecture choice to those that are easy to optimize; the paper asks whether a trained or untrained guide architecture's function can be transferred to a different target architect

Approach

Progressively replace guide layers with target modules, optimizing each stage's replacement to match guide activations via representational similarity (linear CKA or a new differentiable mutual-nearest-neighbors metric, D-MNN), with previously replaced layers jointly re-optimized; then standard end-to-end task training. Works across architecture families (CNN to MLP, ViT to token-wise MLP, Transfo

Key results

• ResNet-18 to MLP with CKA reaches 62.12% ImageNet top-1 vs 33.36% for naive replacement (guide 69.66%).
• GPT-2 to RNN with CKA reaches perplexity 50.58 vs 121.19 for naive replacement (guide 37.50).
• Converting from an untrained ResNet-50 to ResNet-18 gives 71.61% top-1 with CKA vs 68.90% for naive replacement.

Surprising findings

• Untrained guide networks transfer useful inductive biases via NoT, performing comparably to trained guides in several settings.
• Replacing DINOv2 attention with token-wise Patch-MLPs preserves much accuracy, suggesting cross-token communication is less necessary for ImageNet classification.

Limitations

• Converted models still trail guide performance in most settings (e.g., ResNet-18 to MLP 62.12% vs 69.66%; GPT-2 to RNN perplexity 50.58 vs 37.50).
• Progressive replacement is computationally expensive in memory and runtime and each stage requires separate learning-rate and epoch tuning.
• Experiments are limited to ImageNet and Wikitext-103 with a small set of architecture pairs.

Experimental setup

Datasets: ImageNet Wikitext-103

Benchmarks: ImageNet Wikitext-103

Metrics: top-1 accuracy perplexity CKA loss

Closest papers by meaning

• Training the Untrainable: Introducing Inductive Bias via Representational Alignment (NeurIPS 2025 · similarity 0.83)
• Efficient Architecture Search by Network Transformation (AAAI 2018 · similarity 0.83)
• Transporting Task Vectors across Different Architectures without Training (ICML 2026 · similarity 0.83)
• Building Optimal Neural Architectures using Interpretable Knowledge (CVPR 2024 · similarity 0.81)
• Prior Knowledge Guided Neural Architecture Generation (ICML 2025 · similarity 0.81)
• STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.80)