Research map / Large language models / Language, safety and interpretability
HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HELM, a family of fully hyperbolic (Lorentz-model) LLMs, including a mixture-of-curvature-experts variant (HELM-MiCE) and a dense variant (HELM-D), with hyperbolic RoPE, RMSNorm and multi-head latent attention, pretrained on about 5B Wikipedia tokens at 100M and 1B scale.
Problem
Euclidean LLMs do not capture the hierarchical, negatively curved geometry of token embeddings, while prior hyperbolic Transformers use a single fixed curvature, lack essential LLM modules (RoPE, RMSNorm), and scale poorly due to quadratic hyperbolic attention.
Approach
Operate entirely in Lorentz hyperbolic space with per-expert distinct curvatures (Mixture-of-Curvature Experts), and add hyperbolic analogues of RoPE (HoPE), RMSNorm, and multi-head latent attention (HMLA) so that hyperbolic LLMs can be trained at billion-parameter scale.
Key results
- HELM-MiCE at 1B reaches 24.9 average accuracy versus 23.9 for the Euclidean DeepSeekV3 1B, with the best score on all five benchmarks.
- HELM-D at 115M reaches 24.1 average accuracy versus 23.2 for LLaMA 115M, beating it on four of five benchmarks.
- Ablation: HELM-MiCE (24.1 avg) outperforms the constant-curvature MiCE-Const (23.7 avg) at 120M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Euclidean dense LLaMA with fully hyperbolic HELM-D | 115M language model · CommonsenseQA 0-Shot | accuracy (%): 20.9±0.3 -> 20.3±0.2 | Hurt |
| replace Euclidean dense LLaMA with fully hyperbolic HELM-D | 115M language model · HellaSwag 0-Shot | accuracy (%): 25.1±0.3 -> 25.9±0.1 | Helped |
| replace Euclidean dense LLaMA with fully hyperbolic HELM-D | 115M language model · OpenbookQA 0-Shot | accuracy (%): 25.4±0.2 -> 27.1±0.4 | Helped |
| replace Euclidean dense LLaMA with fully hyperbolic HELM-D | 115M language model · MMLU 5-Shot | accuracy (%): 23.4±0.5 -> 25.6±0.2 | Helped |
| replace Euclidean dense LLaMA with fully hyperbolic HELM-D | 115M language model · ARC-Challenging 5-Shot | accuracy (%): 21.0±0.2 -> 21.4±0.3 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 120M language model · CommonsenseQA 0-Shot | accuracy (%): 19.3±0.2 -> 19.7±0.3 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 120M language model · HellaSwag 0-Shot | accuracy (%): 25.3±0.1 -> 25.9±0.2 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 120M language model · OpenbookQA 0-Shot | accuracy (%): 24.0±0.4 -> 27.7±0.4 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 120M language model · MMLU 5-Shot | accuracy (%): 23.9±0.3 -> 24.4±0.2 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 120M language model · ARC-Challenging 5-Shot | accuracy (%): 22.2±0.3 -> 23.2±0.5 | Helped |
| replace Euclidean DeepSeekV3 with hyperbolic HELM-MiCE | 1B language model · MMLU 5-Shot | accuracy (%): 23.6 -> 25.9 | Helped |
| replace distinct expert curvatures with a constant curvature of -1.0 (ablation) | HELM-MiCE 120M · HellaSwag 0-Shot | accuracy (%): 25.6 vs 25.9±0.2 | Hurt |
Surprising findings
- Hyperbolic word embeddings were unstable at 1B scale, so only the space-like dimension of Lorentz embeddings is trained.
- Absolute accuracies are near chance on most benchmarks (roughly 20-28%), so gains of 1-2 points are small in absolute terms.
Limitations
- Experiments use only ~5B Wikipedia tokens and small models, far below commercial LLM scale.
- Euclidean baselines and HELM are not compared against publicly pretrained frontier models.
- Runtime overhead of up to 1.55x and memory overhead of 1.11x relative to Euclidean counterparts.
Experimental setup
Datasets: English Wikipedia RedPajama (token embedding curvature analysis)
Benchmarks: CommonsenseQA HellaSwag OpenbookQA MMLU ARC-Challenging
Metrics: accuracy
Baselines: LLaMA DeepSeekV3 MiCE-Const
Builds on: Hypformer Lorentz hyperbolic neural networks RoPE Multi-Head Latent Attention DeepSeekV3 MoE
Closest papers by meaning
- Hyperbolic Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Language Models as Hierarchy Encoders (NeurIPS 2024 · similarity 0.88)
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
- AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment (NeurIPS 2024 · similarity 0.87)
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.87)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.