Research map / Architectures and efficiency / RNNs and normalization

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically on deep linear networks that residual-connection topology can induce exponential convergence gaps, and proposes ANCRe, which learns softmax-normalized weights over all possible shortcuts to reassign residual connections, yielding faster pre-training for LLaMA, DiT, and deep ResNets RL at <1% overhead.

Problem

Residual connections are almost always placed in a fixed cascaded layout, yet their topology may strongly affect optimization and depth utilization in deep networks; no principled way exists to choose a good topology.

Approach

Parameterize all possible inter-layer shortcuts i:j with learnable coefficients normalized by a softmax over incoming connections (ingoing normalization), so the model learns a data-driven residual topology on the fly with only K(K+1)/2 extra parameters.

Key results

• ANCRe reduces LLaMA validation perplexity by 0.73 ± 0.33 on average over cascaded residuals and matches baseline perplexity with on average 34.3% fewer training iterations.
• On DiT-B/2 with cfg=1.5, ANCRe improves FID from 22.41 to 20.53.
• ANCRe adds negligible overhead: throughput drop <1% and peak memory increase below 0.01GB; 16-layer ResNet with ANCRe matches or surpasses a 4x deeper ResNet in RL.

Surprising findings

• Learnable coefficients over all shortcuts with constant weights (or learnable cascaded residuals) either diverge or underperform, indicating gains come from the combination of topology and adaptive coefficients.
• Frozen learned topology retrained from scratch retains nearly all gains.

Limitations

• Theory limited to deep linear networks under gradient flow with whitened data; nonlinear extension left to future work.
• Benefits appear mainly for deep networks; gains on shallower DiT are smaller.
• Sparse variants needed for gradient checkpointing and pipeline parallelism costs; long-range shortcuts increase recomputation and communication.

Experimental setup

Datasets: C4 ImageNet-1K Gymnasium Humanoid Gymnasium Ant_Big_Maze Gymnasium Arm_Push_Hard Gymnasium Arm_Binpick_Hard

Benchmarks: C4 validation perplexity ImageNet FID-50K Gymnasium goal-reaching tasks time-at-goal

Metrics: perplexity FID sFID inception score precision recall time-at-goal training throughput peak GPU memory

Baselines: cascaded residual connections hyper-connections (HC) manifold-constrained hyper-connections (mHC) GaLore

Builds on: ResNet Transformer hyper-connections GaLore contrastive RL

Closest papers by meaning

• Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.84)
• Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.84)
• Adaptive Depth Networks with Skippable Sub-Paths (NeurIPS 2024 · similarity 0.84)
• Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.84)
• Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.83)
• Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.82)