Research map / Architectures and efficiency / RNNs and normalization
Hyper-Connections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hyper-connections, learnable depth and width connections that expand the residual stream into n hidden copies, to replace residual connections in LLM pretraining and vision models.
Problem
Residual connections (Pre-Norm and Post-Norm) predefine connection strength between layer outputs and inputs, creating a trade-off (seesaw) between gradient vanishing and representation collapse.
Approach
Replace the fixed residual connection with learnable hyper-connections: the hidden state is expanded into n copies with learnable depth-connections (input/output weights per layer) and width-connections (mixing among copies), optionally made input-dependent (dynamic). Pre-Norm and Post-Norm are shown to be special non-trainable cases, and different settings can realize sequential or parallel layer
Key results
- OLMoE-1B-7B-DHC x4 converges about 1.8x faster than the baseline and improves ARC-Challenge by 6 points at 500B tokens.
- OLMo-1B-DHC x4 reduces V2 eval loss from 2.811 to 2.779 and raises downstream average from 62.5 to 64.4.
- On ImageNet, ViT/16-Large with DHC x2 reaches 79.94% top-1 vs 77.25% for the re-implemented baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace residual connections with DHC×4 | OLMo-7B · OLMo-7B V2 Eval | loss: 2.581 -> 2.559 | Helped |
| Replace residual connections with DHC×4 | OLMoE-1B-7B · ARC-Challenge | accuracy: 41.8 -> 47.8 | Helped |
| Replace residual connections with SHC×2 | DiT-XL/2 · ImageNet 256×256 | FID: 2.36 -> 2.18 | Helped |
| Replace residual connections with SHC×2 | ViT/16 Base · ImageNet 224×224 | top-1 accuracy: 76.38 -> 77.60 | Helped |
| Replace residual connections with DHC×2 | ViT/16 Large · ImageNet 224×224 | top-1 accuracy: 77.25 -> 79.94 | Helped |
| Use expansion rate 1 instead of 4 (ablation) | OLMo-1B-DHC · V2 Eval | loss: 2.811 -> 2.819 | Hurt |
| Increase DHC expansion rate from 1 to 4 (ablation) | OLMo-1B · V2 Eval | loss: 2.819 -> 2.781 | Helped |
| Use expansion rate 8 instead of 4 (ablation) | OLMo-1B-DHC · V2 Eval | loss: 2.781 -> 2.778 | No clear effect |
| Use static rather than dynamic hyper-connections at expansion rate 4 (ablation) | OLMo-1B · V2 Eval | loss: 2.781 -> 2.791 | Hurt |
| Remove tanh from DHC×4 (ablation) | OLMo-1B · V2 Eval | loss: 2.781 -> 2.779 | Helped |
| Do not train width connections (ablation) | OLMo-1B-DHC×4 without tanh · V2 Eval | loss: 2.779 -> 2.804 | Hurt |
| Do not train B (ablation) | OLMo-1B-DHC×4 without tanh · V2 Eval | loss: 2.779 -> 2.781 | Hurt |
Surprising findings
- Expansion rate n=1 (a single hidden stream) with dynamic hyper-connections performs worse than the baseline.
- Learned connection patterns show a Lambda-shaped pattern and emergent parallel transformer block-like structures.
Limitations
- Gains for 7B dense models are modest (downstream 70.1 to 71.0).
- n=1 variant fails to improve, so expansion is required; memory overhead grows with expansion rate.
- Vision gains diminish with longer training due to repeated epochs over the same dataset.
Experimental setup
Datasets: dolma-v1.5-sample OLMoE-MIX ImageNet-1k
Benchmarks: C4-en validation loss HellaSwag ARC-Challenge ARC-Easy PIQA WinoGrande BoolQ MMLU Var ImageNet top-1 accuracy ImageNet FID
Metrics: validation loss perplexity zero-shot accuracy top-1 accuracy FID training loss
Baselines: OLMo-1B OLMo-7B OLMoE-1B-7B Altup ResiDual Two-hop residual Parallel transformer block DiT-XL/2 ViT/16
Builds on: Pre-Norm Post-Norm OLMo OLMoE DiT
Closest papers by meaning
- Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.86)
- HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories (CVPR 2025 · similarity 0.85)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.85)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.85)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.85)
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.