Research map / Learning theory and algorithms / Matrix and tensor methods

Spectral-Sphere-Constrained Hyper-Connections

NeurIPS 2026 · Learning theory and algorithms › Matrix and tensor methods · approach: Geometry of low-rank and SPD structure · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes s²HC, which constrains hyper-connection residual matrices to an affine spectral-norm sphere (via a Helmert-basis SVD parameterization with Cayley-transformed orthogonal factors) instead of the doubly stochastic Birkhoff polytope, to avoid identity degeneration and factorial parameterization cost.

Problem

Doubly stochastic constraints on hyper-connection residual matrices (mHC, mHC-lite, KromHC) cause identity degeneration, a spectral expressivity bottleneck, and parameterization inefficiency (unstable Sinkhorn-Knopp iterations or factorial/prime-factor scaling).

Approach

Replace the doubly stochastic constraint with an affine-constrained spectral norm sphere: decompose residual matrix as uniform matrix J plus a zero-marginal part whose spectral norm is bounded by 1, parameterized as orthogonal Helmert-basis factors times a tanh-bounded diagonal of singular values. This gives exact spectral control and free subdominant singular values without Sinkhorn or permutatio

Key results

• On Qwen3-0.6B, s²HC reaches 47.1% average accuracy vs 46.1% for mHC and RC, the best among compared methods.
• On Gemma3-1B, s²HC averages 50.2% vs 50.0% for KromHC and 48.9% for RC; on Qwen2.5-1.5B it averages 50.7% vs 49.6% for mHC-lite.
• mHC's composite mapping column sums deviate across 28 layers (outliers up to 3.6), while s²HC keeps unit column sums and spectral norm near 1.

Surprising findings

• Strongest doubly stochastic variant improves over the plain residual connection by only +0.0 average points on Qwen3, suggesting the gains from prior constrained HC are small.
• Learned residual matrices of mHC, mHC-lite and KromHC concentrate near identity (entries near 0 and 1), i.e., cross-stream mixing is largely inactive.

Limitations

• Experiments limited to 0.6B–1.5B models trained on compute-optimal token budgets; scaling to larger models untested.
• Inherits memory-access overhead of multi-stream hyper-connection architectures, which can limit throughput.
• Gains over baselines are small (often within about 1 point average) and evaluated on a single seed-level comparison without reported variance.

Experimental setup

Datasets: FineWebEdu

Benchmarks: WinoGrande BoolQ PIQA HellaSwag ARC-E ARC-C MMLU TruthfulQA

Metrics: accuracy

Baselines: RC mHC mHC-lite KromHC Hyper-Connections (HC) Orthogonal-constrained HC

Builds on: Hyper-Connections mHC mHC-lite KromHC Sinkhorn-Knopp

Closest papers by meaning

• KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices (ICML 2026 · similarity 0.89)
• Hyper-Connections (ICLR 2025 · similarity 0.84)
• MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation (NeurIPS 2026 · similarity 0.84)
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.83)
• Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds (NeurIPS 2026 · similarity 0.83)
• Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.83)