Research map / Architectures and efficiency / RNNs and normalization
mHC: Manifold-Constrained Hyper-Connections
Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Mingyu Xu, Kuai Yu, Liang Zhao, Shangyan Zhou and 8 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| constrain residual mapping to doubly stochastic manifold (mHC) instead of HC | HC 27B · BBH | EM: 48.9 -> 51.0 | Improved |
| constrain residual mapping (mHC) instead of HC | HC 27B · DROP | F1: 51.6 -> 53.9 | Improved |
| constrain residual mapping (mHC) instead of HC | HC 27B · GSM8K | EM: 53.2 -> 53.8 | Improved |
| constrain residual mapping (mHC) instead of HC | HC 27B · MATH | EM: 26.4 -> 26.0 | Hurt |
| constrain residual mapping (mHC) instead of HC | HC 27B · MMLU | accuracy: 63.0 -> 63.4 | Improved |
| use mHC instead of plain residual baseline | 27B Baseline · 8 downstream benchmarks | mixed metrics: 43.8 -> 51.0 BBH | Improved |
| use mHC instead of HC for training stability | HC 27B · 27B pretraining | gradient norm / loss: final loss reduction of 0.021 | Improved |
| use mHC instead of HC | HC · composite mapping max gain | max backward gain: nearly 3000 -> approximately 1.6 | Improved |
| add learnable H_res mapping to residual stream (ablation) | residual baseline · HC component ablation | absolute loss gap: 0.0 -> -0.022 | Improved |
| add H_pre mapping on top of H_res (ablation) | HC · HC component ablation | absolute loss gap: -0.022 -> -0.025 | Improved |
| add H_post mapping on top of H_res and H_pre (ablation) | HC · HC component ablation | absolute loss gap: -0.025 -> -0.027 | Improved |
| scale from 3B to 27B parameters | mHC · compute scaling | relative loss improvement vs baseline | No clear effect |
Closest papers by meaning
- Spectral-Sphere-Constrained Hyper-Connections (NeurIPS 2026 · similarity 0.84)
- Topological Blindspots: Understanding and Extending Topological Deep Learning Through the Lens of Expressivity (ICLR 2025 · similarity 0.84)
- Hyper-Connections (ICLR 2025 · similarity 0.83)
- MANTRA: The Manifold Triangulations Assemblage (ICLR 2025 · similarity 0.82)
- MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation (NeurIPS 2026 · similarity 0.82)
- ManifoldCache: Training-Free Diffusion Acceleration via Constraint Manifold Caching (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
