Research map / Label-efficient and robust learning / Knowledge distillation

LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LARK, a learnability-grounded trajectory selector that scores teacher reasoning trajectories with a forward-pass proxy of the loss-decay rate and applies a closed-form chi^2-regularized soft top-B weighting for SFT distillation.

Problem

Selecting which teacher-generated reasoning trajectories to use for distilling reasoning into a student, when multiple candidate trajectories exist per question; existing heuristics ignore whether a trajectory is learnable by the student.

Approach

Defines an anchor-time learnability rate rho (squared gradient norm over loss) and approximates its gradient with a forward-pass proxy computed from softmax residuals; a chi^2-regularized objective yields a closed-form soft top-B weighting by score margin above the (B+1)-th score.

Key results

• On Qwen-2.5-7B at B=1, LARK averages 58.01% Acc@5 vs 50.85% for the strongest baseline RSR (+7.16 points).
• LARK achieves the best average in all six model-budget settings, e.g. 41.09% on Qwen-2.5-1.5B and 26.29% on Llama-3.2-3B at B=1.
• Per-teacher mean of the LARK score correlates with downstream AMC accuracy (Pearson r = 0.69, p = 0.018).

Surprising findings

• The inverse loss alone is the weakest trajectory score, indicating likelihood does not characterize learnability.
• Selected trajectories have lower correct-token probability but higher wrong-token concentration, i.e. the student is confidently wrong in a structured way.

Limitations

• Theoretical guarantees rely on NTK lazy-training assumptions and approximations (e.g., near-orthogonal per-trajectory gradients) that hold only approximately in practice.
• Evaluated on a single 5,000-question math pool with 33 candidates per question; extension to noisier pools and broader domains is left open.
• Teacher-level correlation is based on only 11 teachers.

Experimental setup

Datasets: NuminaMath

Benchmarks: AIME-2024 AMC GPQA-Diamond MATH-500

Metrics: Acc@5 mean and std over three decoding seeds

Baselines: Random Token Length Rule-based Quality LLM-judged Quality GRAPE Local Naturalness RSR

Builds on: GRAPE RSR Local Naturalness Polyak-Lojasiewicz condition neural tangent kernel

Closest papers by meaning

• The Signal is in the Steps: Local Scoring for Reasoning Data Selection (ICML 2026 · similarity 0.88)
• Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory? (ICLR 2026 · similarity 0.87)
• Training-Trajectory-Aware Token Selection (ICML 2026 · similarity 0.87)
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching (ICML 2026 · similarity 0.86)
• TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks (ICLR 2026 · similarity 0.86)
• DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling (ICML 2026 · similarity 0.86)