Research map / Large language models / Instruction and fine-tuning
Temperature Scaling in Discrete Sequence (Language) Models
Hannah Scheufele, Peter Blohm, Vikas K Garg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Lk or Lp fine-tuning instead of myopic token-wise scaling | BD3LM · OpenWebText | NLL (T=0.5): 5.28 -> 3.41 (Lk=1) | Improved |
| use Lk or Lp fine-tuning instead of myopic scaling | BD3LM · OpenWebText | schedule variance: 624.3 -> 72.3 (Lp) | Improved |
| use Lp instead of myopic scaling | BD3LM · OpenWebText | Kendall tau: 0.862 -> 0.750 | Hurt |
| use Lp instead of myopic scaling to reach high temperatures | BD3LM · OpenWebText | Teff max: 1.04 -> 2.36 | Improved |
| use Lp instead of myopic scaling | GPT2 · OpenWebText | NLL (T=0.5): 5.50 -> 3.33 | Improved |
| add Ldiff (SSUM objective) to Lp (ablation) | Lp · OpenWebText, target T=0.8 | effective temperature: 0.86 -> 0.90 (GPT2); 0.80 -> 0.93 (BD3LM) | Mixed |
| add Ldiff to Lp (ablation) | Lp · OpenWebText, target T=0.8 | NLL: 3.44 -> 3.07 (BD3LM) | Improved |
| use KL-based objectives LSSTS/LHTS | LSSTS, LHTS · OpenWebText, target T=0.8 | effective temperature: 0.97 (LSSTS), 0.89 (LHTS) vs target 0.8 | No clear effect |
| increase MC samples k from 1 to 4 in Lk (ablation) | Lk · OpenWebText | NLL: 3.41 -> 3.39 | No clear effect |
| fine-tune with Lp, T=0.25 instead of myopic TS | Phi-3.5-mini-instruct · MATH500 | pass@1: 0.446 -> 0.452 | No clear effect |
| fine-tune with Lp, T=0.25 instead of myopic TS | Phi-3.5-mini-instruct · HumanEval | pass@1: 0.035 -> 0.095 | Improved |
| fine-tune with Lp, T=0.25 instead of myopic TS | Qwen2.5-7B · HumanEval | pass@1: 0.539 -> 0.370 | Hurt |
Closest papers by meaning
- Optimizing Temperature for Language Models with Multi-Sample Inference (ICML 2025 · similarity 0.90)
- Improving Diversity in Language Models: When Temperature Fails, Change the Loss (ICML 2025 · similarity 0.88)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.88)
- Parallel Scaling Law for Language Models (NeurIPS 2025 · similarity 0.87)
- OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.87)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
