Research map / Large language models / Language, safety and interpretability
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TempNet, a small network that predicts per-sample temperatures for LLM softmax and CLIP contrastive losses, trained under a KL-constrained DRO robust loss, and shows gains on LLM common-sense/perplexity tasks and CLIP retrieval/zero-shot classification.
Problem
Setting the temperature in softmax-type losses for large foundation models is done by hand or as a single global value, which ignores data heterogeneity; individualized learnable temperatures scale poorly and overfit.
Approach
Derive a KL-DRO robust softmax/contrastive loss with an instance-wise temperature, and replace the per-instance temperatures with a small temperature prediction network (TempNet) that pools prototypical logits, justified by a variational-analysis lemma that the functional minimization equals pointwise minimization.
Key results
- TempNet on fixed LLaMA2 7B improves common-sense average from 57.20 to 59.44 and Lambada perplexity from 4.01 to 3.21.
- On CC3M CLIP training, TempNet reaches 46.17 Flickr30K image-retrieval IR@1 versus 44.36 for iSogCLR, and 42.28 ImageNet1K top-1 versus 40.72.
- TempNet-enabled LLaMA2 Chat 70B reaches 15.83% LC win rate on AlpacaEval versus 14.69% for the leaderboard model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TempNet to LLaMA2 Chat models | LLaMA2 Chat on AlpacaEval · AlpacaEval | Win Rate: 4.96% -> 5.43%; 7.70% -> 7.73%; 13.89% -> 15.05% | Helped |
| Train GPT-2 from scratch with TempNet | GPT-2 · Common Sense Reasoning Average | accuracy: 31.44 -> 32.24 | Helped |
| Finetune GPT-2 with TempNet | GPT-2 · Common Sense Reasoning Average | accuracy: 35.42 -> 36.48 | Helped |
| Train TempNet while keeping LLaMA2 7B fixed | LLaMA2 7B · Common Sense Reasoning Average | accuracy: 57.20 -> 59.44 | Helped |
| Train TempNet while keeping LLaMA2 13B fixed | LLaMA2 13B · Common Sense Reasoning Average | accuracy: 60.00 -> 61.24 | Helped |
| Use TempNet instead of fixed temperatures for contrastive learning | CLIP encoders · Flickr30K Retrieval | IR@1: 44.36$\pm$0.12 -> 46.17$\pm$0.14 | Helped |
| Use TempNet instead of fixed temperatures for contrastive learning | CLIP encoders · MSCOCO Retrieval | IR@1: 23.27$\pm$0.18 -> 24.83$\pm$0.16 | Helped |
| Use TempNet instead of fixed temperatures for contrastive learning | CLIP encoders · ImageNet1K | top-1 accuracy: 40.72$\pm$0.23 -> 42.28$\pm$0.19 | Helped |
| Add TempNet at maj@1, replacing the fixed temperature | LLaMA2 7B · GSM8K | maj@1: 13.2±0.9 -> 12.8±0.9 | Hurt |
| Add TempNet at maj@64, replacing the fixed temperature | LLaMA2 7B · GSM8K | maj@64: 16.5±1.0 -> 17.3±1.0 | Helped |
| Add TempNet instead of default temperature | LLaMA2 Chat on MT-bench · MT-bench, 7B and 70B | score: 6.27 -> 6.44; 6.89 -> 6.97 | Helped |
| Add TempNet instead of default temperature | LLaMA2 Chat 13B · MT-bench | score: 6.65 -> 6.58 | Hurt |
Surprising findings
- DRO loss alone or TempNet alone does not improve much over the standard baseline; only their combination gives notable gains.
- On GSM8K maj@1, TempNet slightly underperforms the fixed tau=0.2 baseline (12.8 vs 13.2).
Limitations
- Experiments on LLMs use a small pretraining corpus (OpenWebText2) as a proof of concept rather than state-of-the-art scale.
- The rho and learning rate hyperparameters are tuned so the mean temperature falls in a target range, and AlpacaEval results required rescaling tau_max at inference.
- TempNet does not clearly help every task (e.g., GSM8K maj@1 and some Wikitext perplexities are slightly worse).
Experimental setup
Datasets: OpenWebText2 CC3M CC12M Flickr30K MSCOCO Lambada Wikitext-2 PIQA HellaSwag ARC-e ARC-c OBQA
Benchmarks: PIQA HellaSwag ARC-e ARC-c OBQA Lambada Wikitext Flickr30K MSCOCO CIFAR10 CIFAR100 ImageNet1K
Metrics: accuracy perplexity recall@k top-1 accuracy win rate LC win rate
Baselines: CLIP CyCLIP SogCLR iSogCLR GPT-2 with tau=1.0 LLaMA2 Chat with default tau=0.7
Builds on: iSogCLR SogCLR KL-DRO CLIP LLaMA2
Closest papers by meaning
- Optimizing Temperature for Language Models with Multi-Sample Inference (ICML 2025 · similarity 0.87)
- Learning to Scale Logits for Temperature-Conditional GFlowNets (ICML 2024 · similarity 0.86)
- Improving Diversity in Language Models: When Temperature Fails, Change the Loss (ICML 2025 · similarity 0.85)
- SEMPO: Lightweight Foundation Models for Time Series Forecasting (NeurIPS 2025 · similarity 0.84)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.84)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.