Research map / Large language models / Instruction and fine-tuning
Hyperbolic Fine-Tuning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLM token embeddings have power-law frequency distributions and low delta-hyperbolicity, then proposes HypLoRA, a LoRA variant that applies low-rank adaptation directly on the Lorentz hyperboloid to improve arithmetic and commonsense reasoning.
Problem
Whether the default Euclidean embedding space suits LLM token representations, given that token embeddings may have hierarchical, tree-like geometry; and how to adapt LLMs in hyperbolic space without the exp/log maps cancelling out.
Approach
Perform LoRA's low-rank transformation directly on the Lorentz hyperboloid (Direct Lorentz Low-Rank Transformation) by recomputing the time-like component, instead of mapping to the tangent space where exp and log maps cancel, so hyperbolic geometry is preserved while adapting Euclidean LLMs.
Key results
- On Qwen2.5-7B, HypLoRA reaches 88.3% arithmetic M.AVG vs 80.8% for LoRA (same 0.71% params).
- On Gemma3-4B commonsense reasoning, HypLoRA averages 82.5% vs LoRA's 79.5%.
- On LLaMA3-8B commonsense reasoning, HypLoRA averages 84.8% vs LoRA's 80.8%; on LLaMA-7B arithmetic, HypLoRA M.AVG 44.4% vs LoRA 43.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use direct hyperbolic low-rank adaptation instead of LoRA | LLaMA-7B · Arithmetic reasoning (M.AVG) | accuracy: 43.7 -> 44.4 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | LLaMA-13B · Arithmetic reasoning (M.AVG) | accuracy: 51.0 -> 51.5 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | Gemma-7B · Arithmetic reasoning (M.AVG) | accuracy: 68.6 -> 71.2 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | LLaMA3-8B · Arithmetic reasoning (M.AVG) | accuracy: 71.9 -> 74.2 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | Gemma3-4B · Arithmetic reasoning (M.AVG) | accuracy: 73.7 -> 77.8 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | Qwen2.5-7B · Arithmetic reasoning (M.AVG) | accuracy: 80.8 -> 88.3 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | LLaMA3-8B · Commonsense reasoning (AVG) | accuracy: 80.8 -> 84.8 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | Gemma3-4B · Commonsense reasoning (AVG) | accuracy: 79.5 -> 82.5 | Helped |
| Use direct hyperbolic low-rank adaptation instead of LoRA | Qwen2.5-7B · Commonsense reasoning (AVG) | accuracy: 85.2 -> 87.0 | Helped |
Surprising findings
- On LLaMA-7B arithmetic, HypLoRA's gain over LoRA is small (44.4 vs 43.7), and on MAWPS HypLoRA is lower than LoRA for several models (e.g., 79.0 vs 81.9 on LLaMA-7B).
- Power-law exponent alone does not guarantee hierarchical embedding; the paper notes hierarchy depends on training objectives.
Limitations
- Hyperbolicity analysis limited to arithmetic and commonsense datasets; generalization to code or multilingual text untested.
- Frequency-norm relationship is correlational, not causal; delta-hyperbolicity measured at prompt level.
- Inference-time overhead from hyperbolic projections, larger for bigger models.
Experimental setup
Datasets: Math10K Commonsense170K GSM8K MAWPS MAWPS-single AQuA SVAMP BoolQ PIQA SIQA HellaSwag WinoGrande
Benchmarks: GSM8K AQuA MAWPS SVAMP BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: accuracy micro-average accuracy average accuracy relative delta-hyperbolicity
Baselines: LoRA DoRA GPT-3.5
Builds on: LoRA hyperbolic neural networks DoRA Math10K / Commonsense170K setup (LLM-Adapters)
Closest papers by meaning
- HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts (NeurIPS 2025 · similarity 0.91)
- Language Models as Hierarchy Encoders (NeurIPS 2024 · similarity 0.87)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.87)
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.