Research map / Large language models / Language, safety and interpretability

The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that overfitting pre-trained LLMs (and ImageGPT) to near-zero training loss on a tiny dataset ("hyperfitting") sharply improves greedy-decoding open-ended generation, as judged by humans and diversity metrics.

Problem

Greedy decoding from pre-trained LLMs produces repetitive, degenerate long-form text; existing fixes are sampling heuristics and repetition penalties, and the cause is poorly understood.

Approach

Fine-tune a pre-trained model with a small learning rate on about 2000 sequences for 20 epochs until training loss is near zero. This "hyperfitting" makes greedy decoding less repetitive and preferred by humans, even though validation perplexity worsens, and the sharpened low-entropy predictions are hypothesized to reflect top-rank encouragement.

Key results

• Hyperfitted TinyLlama 1.1B raises human preference at 256 tokens from 4.9% to 34.4%, on par with Llama 3.1 70B (34.4%).
• Hyperfitted Llama 3.1 70B reaches 52.4% 256-token preference, versus 34.4% for the original 70B model.
• Hyperfitted models produce much lower-entropy predictions; Llama 3.1 8B @1 probability rises from 48.4% to 74.4%.

Measured effects reported in the paper

Surprising findings

• Greedy decoding from hyperfitted models outperforms Top-P sampling from the original models in human preference and TTR.
• Explicitly blocking training-data subsequences does not noticeably reduce output quality, and hyperfitted models rarely re-cite training data.

Limitations

• Validation perplexity becomes very poor (e.g., 94.7 to 545 vs 9-36 originally), so perplexity does not capture generation quality.
• Effects of the training data choice are inconclusive; shuffling the same data changes ~30% of top-1 predictions.
• Image generation results are preliminary and far below diffusion models in quality.

Experimental setup

Datasets: Fiction-Stories Wikipedia BBC News CIFAR-10 MMLU GLUE

Benchmarks: MMLU GLUE

Metrics: human preference ratio type-token ratio perplexity Self-BLEU dataset BLEU dataset overlap accuracy

Baselines: Top-P sampling original pre-trained models TinyLlama 1.1B Top-P DeepSeek 7B Top-P Llama 3.1 8B Top-P

Builds on: nucleus sampling grokking double descent

Closest papers by meaning

• Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.88)
• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.88)
• Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.88)
• LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.88)
• Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation (NeurIPS 2025 · similarity 0.88)
• When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.88)