Research map / Large language models / Instruction and fine-tuning

Training ML Models with Predictable Failures

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes the forecast error of the Gumbel-tail (OLS on top-k order statistics) extrapolation method for deployment-scale failure rates and fine-tunes models with a differentiable "forecastability loss" so their failure tails become well-described by the forecaster, validated on a Qwen3-0.6B password game and an 8x8 RL gridworld.

Problem

Extrapolating from a small evaluation set to predict deployment-scale worst-case failure rates is unreliable when the failure tail is not log-linear or when rare high-failure modes are absent from the fit set; existing work treats the model as fixed.

Approach

Derive a finite-k decomposition of the Gumbel-tail forecast error into rank, curvature, occupancy and remainder terms, then use the (differentiable) squared forecast error across extrapolated ranks as a fine-tuning loss, with per-step random partitioning, a union top-C cache, and an improving-only gradient mask to couple forecast accuracy with safety.

Key results

• On the Qwen3-0.6B password game, forecastability training reaches ~60x forecast-error improvement over the pretrained baseline (~75x with calibration), while SFT, calibration, and SFT+calibration plateau below 20x.
• Forecastability training matches SFT on worst-rank held-out leak probability (~40 decades below baseline) but yields over two orders of magnitude lower WildChat KL drift.
• In the gridworld, Ours reaches 14x forecast improvement vs SFT 15x; with calibration Ours+cal reaches 52x vs SFT+cal 26x, and worst-case regret falls to 1/3.5 vs SFT 1/2.6.

Surprising findings

• The Gumbel-tail forecast has a built-in overprediction bias (E[xi] > 0) for essentially all practical (k, R), which is safety-favorable.
• Fixed fit/deploy partitions during fine-tuning cause an asymmetry-overfit failure, requiring per-step re-partitioning.

Limitations

• Experiments are small proof-of-concepts (0.6B LM with LoRA, 8x8 gridworld); 8B run has only three seeds and no SFT comparator.
• RL setting relies on closed-form regret from backward value iteration; Monte Carlo regret estimation is not addressed.
• Bulk-plus-rare-mode mixtures are synthetic; natural rare modes untested.

Experimental setup

Datasets: WildChat-1M

Benchmarks: password game (single-token leak probability) 8x8 gridworld navigation with traps

Metrics: worst-rank leak probability (decades reduction) WildChat single-token KL divergence fold improvement in worst-rank squared log-prob forecast err pre-training mean return worst-case held-out regret

Baselines: Cal. (post-hoc affine calibration) SFT SFT+cal.

Builds on: Gumbel-tail method (Jones et al. 2025) GCG LoRA conformal risk control

Closest papers by meaning

• Routing Subspaces: Auditing Evaluation-to-Deployment Mismatch in Fine-Tuned Language Models (NeurIPS 2026 · similarity 0.84)
• Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias (NeurIPS 2026 · similarity 0.84)
• Efficient Prediction of Pass@k Scaling in Large Language Models (NeurIPS 2026 · similarity 0.83)
• Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.83)
• Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models (ICML 2026 · similarity 0.83)
• Reliably detecting model failures in deployment without labels (NeurIPS 2025 · similarity 0.83)