Research map / Large language models / Language, safety and interpretability

Fast and Accurate Probing of In-Training LLMs' Downstream Performances

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes lightweight probes that read LLM checkpoint hidden states to predict per-prompt pass@1 on downstream tasks, avoiding costly generative evaluation during training, validated on OLMo-3-7B checkpoints.

Problem

In-training evaluation of LLM downstream performance via generation is too slow and costly as test-time compute grows, while training loss/perplexity does not reliably track downstream task success.

Approach

Treat per-prompt pass@1 as a value to regress and train lightweight probes (a submodel probe over base-model hidden states, or LoRA adapters) that predict success probability from internal representations, enabling forward transfer across later checkpoints without generation.

Key results

• Submodel probe reaches average AUROC 0.7890 on pre-training checkpoints vs 0.5795 for loss fit and 0.5810 for linear probe.
• Submodel probe trained at step 200k keeps AUROC around 0.75 on the final checkpoint, while LoRA probe drops from 0.67 to 0.56.
• Probe evaluation gives a 15.6x speedup per checkpoint on base model (0.78h to 0.05h), and up to 231.8x on the think model.

Surprising findings

• Loss-based fit barely outperforms random guessing (AUROC 0.5795), indicating training loss poorly tracks downstream reasoning success.
• Probes reach above 0.6 AUROC on hard AIME and GPQA benchmarks despite requiring very long generations.

Limitations

• Probes are trained and tested on the same downstream tasks with per-prompt train/test splits; generalization to unseen tasks is not shown.
• MMLU performance is poor (AUROC around 0.67) attributed to limited training data (10k samples subset).
• Probe preparation requires full generative evaluation on training split, which can take many hours for think models (112h data collection).

Experimental setup

Datasets: MMLU GSM8K MATH BBH HumanEval MBPP AIME GPQA

Benchmarks: MMLU GSM8K MATH BBH HumanEval MBPP AIME GPQA

Metrics: AUROC MSE absolute error speedup

Baselines: Loss fit Linear probe Subset sampling

Builds on: LoRA linear probing

Closest papers by meaning

• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.89)
• Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.89)
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.89)
• LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.89)
• The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.89)