Research map / Large language models / Instruction and fine-tuning
Fresh in memory: Training-order recency is linearly encoded in language model activations
Dmitrii Krasheninnikov, Richard E. Turner, David Krueger
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune 30 extra epochs on shuffled mixed data from all stages (ablation) | Llama-3.2-1B probes · D1 vs D6 probing (synthetic) | probe accuracy (%): 90 -> 63 | Hurt |
| fine-tune 30 extra epochs on shuffled mixed data from all stages (ablation) | Llama-3.2-1B probes · D1 vs D6 probing (natural) | probe accuracy (%): 80 -> 60 | Hurt |
| use LoRA on Llama-3.1-8B instead of full fine-tuning (ablation) | Llama-3.1-8B · D1 vs D6 probing | probe accuracy (%): ~85 | No clear effect |
| use 1-epoch stages with 5x more unique examples instead of 5 epochs (ablation) | Llama-3.2-1B natural data · D1 vs D6 probing | probe accuracy (%) | No clear effect |
| re-expose one earlier stage as a 7th fine-tuning stage (ablation) | Llama-3.2-1B · activation centroid order | centroid position | Improved |
| train one stage for 15 epochs instead of 5 (ablation) | Llama-3.2-1B · activation centroid order | centroid gap | Improved |
| balance joint distribution of six activation statistics between D1 and D2 (ablation) | Llama-3.2-1B probes · D1 vs D2 probing | probe accuracy (%) | No clear effect |
| balance joint distribution of seven logit statistics (ablation) | Llama-3.2-1B probes · D1 vs D2 probing | probe accuracy (%) | No clear effect |
| balance forward-looking multi-token statistics (ablation) | Llama-3.2-1B probes · D1 vs D2 probing | probe accuracy (%) | Mixed |
| fine-tune to report training stage explicitly | Llama-3.2-1B (D1 then D2) · held-out aliases | accuracy (%): 79.8 | Improved |
| probe specific training datapoints instead of held-out entities | Llama-3.2-1B · question-level stage probing | probe accuracy (%): over 90 -> ~60 | Hurt |
Closest papers by meaning
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.88)
- Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · similarity 0.88)
- Emergence of Linear Truth Encodings in Language Models (NeurIPS 2025 · similarity 0.88)
- On Linear Representations and Pretraining Data Frequency in Language Models (ICLR 2025 · similarity 0.88)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.87)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
