Research map / Large language models / Agents, code and math
EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EvaLearn, a benchmark of 648 challenging problems in 182 sequences that evaluates LLM learning capability and efficiency by having models solve same-task problems sequentially with feedback or demonstrations, scored by rubric-based GPT-4o judging.
Problem
Existing LLM benchmarks evaluate models in a parallel, i.i.d. fashion and measure only static ability, leaving a model's capacity to learn from experience within a task and the speed of that learning unmeasured.
Approach
Replace parallel evaluation with sequential solving: each sequence has seven same-type problems that models solve in order, with prior problems, canonical answers, or judge feedback in context, and five metrics (accuracy, fitted slope, first-correct position, first-learned offset, consecutive correct streak, post-warmup accuracy) quantify learning.
Key results
- OpenAI-o3-mini gains +10.5 points overall under feedback learning (54.3% to 64.8%), the largest improvement among models.
- Claude-3.7-Sonnet shows the highest learning efficiency with slope k=2.08, while DeepSeek-R1 drops 9.3 points under feedback learning.
- Judge-to-human validation accuracy exceeds 95% across all tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sequential feedback learning instead of zero-shot solving | OpenAI-o3-mini · EvaLearn overall | accuracy: 54.3 -> 64.8; +10.5% | Helped |
| use sequential feedback learning instead of zero-shot solving | Claude-3.7-Sonnet · EvaLearn overall | accuracy: 28.4 -> 35.6; +7.2% | Helped |
| use sequential feedback learning instead of zero-shot solving | Claude-3.7-Sonnet-Thinking · EvaLearn overall | accuracy: 31.2 -> 37.4; +6.2% | Helped |
| use sequential feedback learning instead of zero-shot solving | DeepSeek-R1 · EvaLearn overall | accuracy: 55.7 -> 46.4; -9.3% | Hurt |
| use sequential feedback learning instead of zero-shot solving | Doubao-1.5-Thinking-Pro · EvaLearn overall | accuracy: 57.1 -> 51.6; -5.5% | Hurt |
| use sequential feedback learning instead of zero-shot solving | feedback-learning models · EvaLearn summarization | accuracy | Hurt |
| use sequential feedback learning instead of zero-shot solving | GPT-4o · EvaLearn mathematical reasoning | accuracy: +18.0% | Helped |
| use sequential feedback learning instead of zero-shot solving | Claude-3.7-Sonnet-Thinking · EvaLearn classification | accuracy: +13.5% | Helped |
| use sequential feedback learning instead of zero-shot solving | OpenAI-o3-mini · EvaLearn logical reasoning | accuracy: +14.1% | Helped |
| provide prior problems and canonical answers rather than three fixed few-shot demonstrations | OpenAI-o3-mini · EvaLearn logical reasoning | accuracy: 50.7 -> 53.1 | Helped |
| provide rubric feedback on prior solutions rather than prior problems and canonical answers | OpenAI-o3-mini · EvaLearn logical reasoning | accuracy: 53.1 -> 59.9 | Helped |
| use sequential feedback learning rather than zero-shot solving | models across EvaLearn tasks · EvaLearn overall | accuracy | Mixed |
Surprising findings
- Stronger static models do not consistently show better learning; DeepSeek-R1 outperforms Claude-3.7-Sonnet zero-shot yet declines under feedback while Claude improves.
- Seven of nine models perform worse with feedback on summarization, suggesting experience can interfere with pre-trained knowledge use.
Limitations
- Limited to problems that can be evaluated with well-defined rubrics, excluding open-ended tasks.
- Benchmarks mainly frontier closed models; limited open-source model coverage.
- Only two learning methods (demonstration and feedback) at inference time were explored.
Experimental setup
Datasets: EvaLearn
Benchmarks: EvaLearn
Metrics: accuracy slope of fitted accuracy curve average position of first correct solution average offset of first learned correct solution average consecutive correct solutions post-warmup accuracy
Baselines: Claude-3.7-Sonnet DeepSeek-R1 DeepSeek-V3 GPT-4o OpenAI-o3-mini Doubao-1.5-Pro Doubao-1.5-Thinking-Pro Qwen2.5-32b-Instruct Gemini-2.5-Pro Gemini-2.5-Flash
Builds on: in-context learning LLM-as-a-judge
Closest papers by meaning
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.91)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.91)
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.90)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.90)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.90)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.