Research map / Large language models / Instruction and fine-tuning
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains PandaLM, a LLaMA-based judge LLM that compares responses from instruction-tuned models, and uses it to select better fine-tuning hyperparameters, validated against a human-annotated test set.
Problem
Hyperparameter selection for instruction tuning LLMs is hard because evaluation is costly, subjective, and often relies on accuracy metrics or privacy-risky API judges such as GPT-4.
Approach
A locally deployable judge LLM, trained on GPT-3.5-distilled pairwise comparisons (with reasons and reference responses) filtered for position-swap consistency, judges which of two fine-tuned models is better, and is used to search hyperparameters for instruction tuning.
Key results
- PandaLM-7B reaches 59.26% accuracy against human labels, comparable to GPT-3.5 (62.96%) and GPT-4 (66.47%); PandaLM-70B reaches 66.87% accuracy and 69.23 F1.
- Hyperparameters selected by PandaLM-7B improve all five base models over Alpaca defaults under GPT-4 judging (e.g., LLaMA-7B 40 wins / 17 losses / 113 ties).
- Average lm-eval score for LLaMA rises from 0.4879 to 0.5031 with PandaLM-selected tuning, but Cerebras-GPT drops from 0.3836 to 0.3752.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PandaLM-selected hyperparameters instead of Alpaca's default hyperparameters | LLaMA-7B, Bloom-7B, Cerebras-GPT-6.7B, OPT-7B, and Pythia-6.9B · 170-instruction validation set | pairwise preference: GPT-4: (40,17,113); (44,34,92); (60,20,90); (39,30,101); (52,30,88) | Helped |
| Include evaluation reasons and reference responses alongside the evaluation label (ablation) | PandaLM-7B · Human-annotated test set | accuracy: 0.4725 -> 0.5926 | Helped |
| Include evaluation reasons and reference responses alongside the evaluation label (ablation) | PandaLM-7B · Human-annotated test set | F1: 0.3152 -> 0.5456 | Helped |
| Scale the judge model from 7B to 70B | PandaLM · LSAT | accuracy: 0.4717 -> 0.6604 | Helped |
| Scale the judge model from 7B to 70B | PandaLM · PubMedQA | accuracy: 0.6154 -> 0.7692 | Helped |
| Scale the judge model from 7B to 70B | PandaLM · BioASQ | accuracy: 0.5152 -> 0.7727 | Helped |
| Use full fine-tuning instead of LoRA (ablation) | Bloom, Cerebras-GPT, LLaMA, OPT, and Pythia · 170-instance pairwise comparisons | wins: Bloom: LoRA 35, Full Fine-tuning 66; Cerebras-GPT: LoRA 40, Full Fine-tuning 59; LLaMA: LoRA 28, Full Fine-tuning 48; OP | Helped |
| Increase instruction-tuning data up to the model-specific reported optimal size (ablation) | OPT · Instruction tuning | performance: 992,000 | No clear effect |
| Use PandaLM-selected hyperparameters instead of Alpaca hyperparameters | LLaMA-7B · lm-eval average | average score: 0.4879 -> 0.5031 | Helped |
| Use PandaLM-selected hyperparameters instead of Alpaca hyperparameters | Cerebras-GPT-6.7B · lm-eval average | average score: 0.3836 -> 0.3752 | Hurt |
Surprising findings
- Lower perplexity does not imply better instruction-tuned performance; LLaMA-PandaLM has higher perplexity than LLaMA-Alpaca yet wins more comparisons.
- Cerebras-GPT declines on lm-eval traditional tasks after PandaLM tuning even though judges prefer it.
Limitations
- Hyperparameter search space is limited to common practice and may miss the optimum.
- Training data derived from GPT-3.5 may not fully match human preferences.
- Only supervised training studied; semi-supervised, noisy, and imbalanced settings not explored.
Experimental setup
Datasets: Alpaca 52K self-instruct human evaluation dataset LSAT PubMedQA BioASQ lm-eval tasks
Benchmarks: PandaLM human-annotated test set LSAT PubMedQA BioASQ lm-eval (ARC, HellaSwag, MMLU, TruthfulQA)
Metrics: accuracy precision recall F1 win/tie/lose counts Cohen's kappa IAA win rate
Baselines: GPT-3.5 GPT-4 BERTScore MAUVE lm-eval Alpaca default hyperparameters
Builds on: Alpaca self-instruct LLaMA DeepSpeed ZeRO LoRA
Closest papers by meaning
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.91)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.91)
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.90)
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · similarity 0.89)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.89)
- SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.