Research map / Large language models / Instruction and fine-tuning
Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transformer Copilot, a Pilot-Copilot framework where an auxiliary Copilot model learns from a logged Mistake Log of the Pilot's fine-tuning signals and rectifies Pilot logits at inference.
Problem
Fine-tuned LLMs often remain suboptimal at inference due to train-test misalignment and overfitting; the model's own learning signals from SFT are discarded after each update.
Approach
Record per-token Pilot signals (inputs, hidden states, token-level prediction errors) into a Mistake Log during SFT, train a Copilot to regress these errors with an RMSE loss, and at inference add lambda times the Copilot output to the Pilot's probabilities to rectify next-token generation.
Key results
- T-Copilot improves Pilot models by up to 34.5% relative (FLAN-T5-small arithmetic average 11.3 to 15.2).
- LLaMA-3.2-3B + T-Copilot-3B (6B total) reaches 82.3 commonsense average vs 82.1 for LLaMA-3.1-8B.
- Qwen2.5-7B + T-Copilot-3B (10B total) averages 88.2 commonsense vs 87.4 for Qwen2.5-14B; reports about 4% average time overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add T-Copilot to the Pilot during fine-tuning and inference | FLAN-T5-small · Arithmetic reasoning Avg. | accuracy: 11.3 -> 15.2; ↑ 34.5% | Helped |
| Add T-Copilot-1B | LLaMA-3.2-3B · Commonsense reasoning Avg. | accuracy: 79.1 -> 80.7; ↑ 2.0% | Helped |
| Add T-Copilot-1B | LLaMA-3.2-3B · Arithmetic reasoning Avg. | accuracy: 59.7 -> 63.2; ↑ 5.9% | Helped |
| Increase Copilot scale from 1B to 3B | LLaMA-3.2-3B · Arithmetic reasoning Avg. | accuracy: 63.2 -> 66.5 | Helped |
| Add T-Copilot-3B | Qwen2.5-7B · Commonsense reasoning Avg. | accuracy: 83.4 -> 88.2; ↑ 5.8% | Helped |
| Add T-Copilot-3B | Qwen2.5-7B · Arithmetic reasoning Avg. | accuracy: 78.1 -> 81.7; ↑ 4.6% | Helped |
| Replace a larger standalone model with a smaller Pilot plus T-Copilot | LLaMA-3.2-3B + T-Copilot-3B vs LLaMA-3.1-8B · Arithmetic reasoning Avg. | accuracy: 65.9 -> 66.5 | Helped |
| Replace a larger standalone model with a smaller Pilot plus T-Copilot | Qwen2.5-7B + T-Copilot-3B vs Qwen2.5-14B · Commonsense reasoning Avg. | accuracy: 87.4 -> 88.2 | Helped |
| Transfer a trained T-Copilot-1B to a new independently trained Pilot without joint training | LLaMA-3.2-1B · Overall reasoning tasks | accuracy improvement: ↑ 6.1% | Helped |
| Add Copilot computation | Transformer Copilot · Inference and training efficiency | time overhead: 4% | Hurt |
Surprising findings
- A Copilot need not match the Pilot's standalone accuracy to improve it, per the theoretical remark.
- Transferred Copilot to new independently trained Pilots of the same architecture still gives gains (e.g., 6.1% overall with LLaMA-3.2-1B).
Limitations
- Gains on some tasks are small (e.g., 2% for some Pilots), and the approach relies on the Pilot's full training trajectory being logged.
- Mistake Log buffer keeps only the most recent 128 training rounds by default.
Experimental setup
Datasets: Commonsense170K Math10K PIQA WinoGrande HellaSwag BoolQ SIQA OpenbookQA AQuA GSM8K MAWPS SVAMP
Benchmarks: PIQA WinoGrande HellaSwag BoolQ SIQA OpenbookQA AQuA GSM8K MAWPS SVAMP Beauty LastFM
Metrics: accuracy Hit@K NDCG@K throughput training time overhead
Baselines: LLaMA-3.1-8B Qwen2.5-14B Gemma-2-9B Mistral-Pro-8B LLaMA-Pro-8B Ministral-8B MergeKit-9B TIES
Builds on: LLaMA-3 Qwen2.5 FLAN-T5 Logit Lens LLaMA-Pro
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.87)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.87)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.87)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.87)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.