Research map / Large language models / Instruction and fine-tuning
Bits That Count: Quantifying and Predicting Capabilities of Language Models
Elizabeth Donoway, Hailey Joren, Michael R DeWeese, Ethan Perez, John Schulman, Fabien Roger, Jan Leike
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-teach multiplication with operator notation before fine-tuning (ablation) | TinyStories-1B LoRA · Multiplication | capacity threshold EDL/P*: 1.02 -> 0.05 bits/parameter | Improved |
| pre-teach a different operation (addition/subtraction) before multiplication (ablation) | TinyStories-1B LoRA · Multiplication | capacity threshold EDL/P*: 1.56 bits/parameter, still increasing returns | No clear effect |
| pre-elicit format with a single related example (ablation) | Llama 3.2 1B · Addition/subtraction | EDL per token: reduces EDL/D by approximately an order of magnitude | Improved |
| fine-tune on a single random example instead of using the instruct model | Llama 3.1 8B base · Addition/subtraction | accuracy: 0% -> 96% (Instruct 56.1%) | Improved |
| scale model size (ablation) | Llama 3 1B/3B/8B · Multiplication, addition/subtraction | capacity threshold EDL/P*: larger models have lower thresholds | Improved |
| use LoRA below capacity threshold vs above (ablation) | LoRA · Elicitation tasks | capacity vs full fine-tuning: ~0.01-0.1 bits/parameter for elicitation vs ~1 for teaching | Hurt |
| distill R1 reasoning traces by SFT | Qwen2.5 models · multi-domain reasoning | accuracy: up to 23 pp from single example | Improved |
| replace EDL with plain MDL (ablation) | capability regime analysis · elicitation vs teaching classification | discrimination | Hurt |
| scale dataset to millions of examples | Llama models · Multiplication | EDL signature: elicitation-like to teaching-like crossover | Mixed |
Closest papers by meaning
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.88)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws (ICLR 2025 · similarity 0.88)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.87)
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
