Research map / Large language models / Agents, code and math
LILO: Learning to Reason at the Frontier of Learnability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LILO, which prioritises RL training on questions with high learnability (success variance) via rejection sampling, with theory linking expected policy improvement to learnability; it speeds up PPO, VinePPO and GRPO training for LLM reasoning.
Problem
Policy-gradient RL for LLM reasoning wastes compute on questions that are always solved or never solved, since they yield zero advantage and no learning signal.
Approach
Estimate each candidate question's learnability p(1-p) from a few sampled attempts, then train only on the top-learnability questions in each batch; a theorem shows expected policy improvement is upper-bounded by a quantity increasing linearly with learnability.
Key results
- PPO+LILO reaches PPO's best MATH test accuracy in 2.5x fewer steps, and VinePPO+LILO reaches the best VinePPO baseline accuracy on MATH in 3.2x fewer steps.
- VinePPO+LILO improves GSM8K final test accuracy from 53.2% to 55.9% (speed-up 3.3x).
- GRPO+LILO on ORZ57K reaches the best GRPO accuracy in 1.5x fewer steps (35.5 to 37.1 final test accuracy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prioritise high-learnability questions (rejection sampling, LILO) during training | PPO · MATH and GSM8K | final test accuracy and speed-up in steps: MATH 19.1→21.8 (2.5x fewer steps); GSM8K 51.1→53.2 (1.9x fewer steps) | Helped |
| prioritise high-learnability questions (LILO) during training | VinePPO · MATH and GSM8K | final test accuracy and speed-up in steps: MATH 22.8→24.9 (3.2x); GSM8K 53.2→55.9 (3.3x) | Helped |
| prioritise high-learnability questions (LILO) during training | GRPO · ORZ57K, averaged over 5 test sets | final test accuracy and speed-up in steps: 35.5→37.1 (1.5x fewer steps) | Helped |
| prioritise high-learnability questions (LILO) during training | PPO and VinePPO trained on MATH, evaluated out-of-distribution · CollegeMath and OlympiadBench | test accuracy@1: CollegeMath +1.0 (PPO), +1.9 (VinePPO); OlympiadBench +0.3 (PPO), +0.4 (VinePPO) | Helped |
| increase candidate pool |D| from 4x to 8x batch size for LILO on VinePPO (ablation) | VinePPO + LILO on GSM8K · GSM8K | test accuracy (%): further improved over |D|=4x (exact value not given in text) | Helped |
| apply LILO to an already heavily RL-tuned model (Oat-Zero-1.5B) on its existing dataset (ablation) | GRPO + LILO on Oat-Zero-1.5B · training on existing dataset | test accuracy: only minimal improvement | No clear effect |
| GRPO with LILO on AIME test set | GRPO + LILO · AIME (ORZ57K-trained) | test accuracy | No clear effect |
| maintain a buffer of high-learnability questions refreshed only periodically (SFL-style) mixed with uniform batches (ablation) | PPO/GRPO-style LLM training · ORZ57K-style buffer experiments | training overfitting and generalisation | Hurt |
Surprising findings
- Learnability prioritisation improves train accuracy faster but the train-test ratio is unchanged, so generalisation gap is not reduced.
- Applying LILO to the already heavily RL-tuned Oat-Zero-1.5B yields only minimal improvement despite finding highly learnable batches.
Limitations
- Learnability ignores aleatoric uncertainty, so inherently random questions may score high.
- LILO is stateless and discards extra learnable questions beyond the batch size, adding sampling overhead.
- Gains on final accuracy are modest and do not close the train-test generalisation gap; results limited to small 1-1.5B models and binary rewards.
Experimental setup
Datasets: GSM8K MATH ORZ57K CollegeMath OlympiadBench Minerva AMC AIME
Benchmarks: MATH GSM8K CollegeMath OlympiadBench Minerva AMC AIME
Metrics: Pass@1 accuracy training speed-up in steps
Baselines: PPO VinePPO GRPO Uniform question sampling (no learnability)
Builds on: VinePPO RLOO GRPO PPO Sampling for Learnability (SFL)
Closest papers by meaning
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.89)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.89)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.88)
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.88)
- D²Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.