Research map / Large language models / Instruction and fine-tuning
Train-before-Test Harmonizes Language Model Rankings
Guanhua Zhang, Ricardo Dominguez-Olmedo, Moritz Hardt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune each model on the benchmark train set before testing | 61 LLMs, 24 benchmarks · Cross-benchmark ranking agreement | mean Kendall tau: 0.52 -> 0.76 | Improved |
| fine-tune before testing, ties by statistical significance | 61 LLMs · Cross-benchmark ranking agreement | mean Kendall tau-b: 0.58 -> 0.77 | Improved |
| fine-tune before testing | NQ-Open outlier benchmark · Agreement with other benchmarks | mean Kendall tau: 0.23 -> 0.74 | Improved |
| fine-tune before testing | language understanding category · Intra-category agreement | mean Kendall tau: 0.52 -> 0.75 | Improved |
| fine-tune before testing | math category · Intra-category agreement | mean Kendall tau: 0.55 -> 0.84 | Improved |
| fine-tune before testing | perplexity vs downstream rankings · Perplexity-downstream agreement | mean Kendall tau: 0.48 -> 0.74 | Improved |
| fine-tune before testing | average perplexity vs average downstream · Aggregate agreement | Kendall tau: 0.55 -> 0.84 | Improved |
| fine-tune before testing | model score matrix, all models · PCA | PC1 explained variance: 70% -> 86% | Improved |
| restrict to Qwen family and fine-tune before testing | Qwen score matrix · PCA | PC1 explained variance: 86% -> 93% | Improved |
| fine-tune before testing | perplexity-perplexity agreement across corpora · Perplexity ranking agreement | mean Kendall tau: 0.76 -> 0.78 | No clear effect |
| use instruction-tuned instead of base models for perplexity predicting potential | pre-fine-tuning perplexity · Post-fine-tuning downstream rankings | mean Kendall tau: 0.78 -> 0.51 | Hurt |
| account for statistical significance (ties) in direct evaluation | direct evaluation rankings · Cross-benchmark ranking agreement | mean Kendall tau: 0.52 -> 0.58 | Improved |
Closest papers by meaning
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.91)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.91)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.90)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- Aligning Language Model Benchmarks with Pairwise Preferences (NeurIPS 2026 · similarity 0.89)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
