Research map / Large language models / Instruction and fine-tuning
Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
Meimingwei Li, Yuanhao Ding, Esteban Garces Arias, Christian Heumann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| hyperfit the model (vs temperature-scaling to match entropy) | TinyLlama-1.1B · greedy generation | TTR: 0.397 -> 0.684 | Improved |
| hyperfit the model (vs temperature-scaling to match entropy) | TinyLlama-1.1B · greedy generation | bigram repetition: 0.604 -> 0.140 | Improved |
| lower temperature to T=0.59 to match hyperfitted entropy (ablation) | TinyLlama-1.1B · greedy generation | TTR: 0.400 -> 0.397 | No clear effect |
| inject static logit bias, alpha=0.01 (ablation) | TinyLlama-1.1B · static injection ablation | bigram repetition: 0.588 -> 0.609 | Hurt |
| inject static logit bias, alpha=0.50 (ablation) | TinyLlama-1.1B · static injection ablation | TTR: 0.449 -> 0.215 | Hurt |
| restrict LoRA to last 5 layers (vs Full LoRA) | TinyLlama-1.1B · hyperfitting reproduction | top-1 agreement: 0.523 -> 0.517 | No clear effect |
| restrict LoRA to last 5 layers (vs Full LoRA) | TinyLlama-1.1B · hyperfitting reproduction | TTR: 0.508 -> 0.469 | Hurt |
| restrict LoRA to last 5 layers (vs Full LoRA) | Qwen2.5-1.5B · hyperfitting reproduction | TTR: 0.575 -> 0.591 | Improved |
| restrict LoRA to last 5 layers (vs Full LoRA) | Qwen2.5-1.5B · hyperfitting reproduction | bigram repetition: 0.248 -> 0.213 | Improved |
| restrict LoRA to last 5 layers (vs Full LoRA) | Qwen2.5-1.5B · LLM-as-judge pairwise | win rate (%): 57.3 vs 41.2 | Improved |
| restrict LoRA to last 5 layers (vs Full LoRA) | LLaMA-3.2-3B · training time | speedup: 1.44x -> 1.99x | Improved |
| restrict LoRA to last 5 layers (vs Full LoRA) | Qwen2.5-1.5B and TinyLlama-1.1B · trainable parameters | trainable parameter reduction: 78.3% and 82.7% | Improved |
Closest papers by meaning
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.90)
- Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.87)
- Improving Diversity in Language Models: When Temperature Fails, Change the Loss (ICML 2025 · similarity 0.87)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.86)
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.86)
- Hyperbolic Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
