Research map / Reinforcement learning / RL from feedback

Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ICRPM, a Transformer-based in-context risk predictive model with one-step look-ahead and exponentially decayed sliding-window augmentation to improve model predictive task sampling for robust Meta-RL, domain randomization, and LLM prompt curriculum RL.

Problem

Vanilla risk predictive models in model predictive task sampling suffer from temporal lag (risk estimated under the current policy but used after the update) and sparse optimization histories, causing biased difficulty estimates and rank-flipping of task risks.

Approach

Recasts risk prediction as in-context sequence modeling: a Transformer conditions on the latest history batch to predict next-step task risk (one-step look-ahead), trained on temporally weighted sliding windows of consecutive history batches, with no extra rollouts.

Key results

• On Countdown-4, ICRPM reaches 55.7% accuracy, surpassing GRESO by 1.9 points and MoPPS by 2.8 points.
• On LunarLander, ICRPM improves CVaR_0.9 at convergence by over 40% relative to ERM (as stated).
• On Walker2dVel meta-test CVaR_0.9, ICRPM achieves -1.5 vs -27.9 for MPTS; ICRPM incurs about 1.08x runtime relative to ERM.

Surprising findings

• The MPTS-with-history-augmentation variant remains below ICRPM, suggesting gains come from look-ahead context conditioning rather than history reuse alone.

Limitations

• Theoretical bounds rely on assumptions such as Lipschitz/smooth losses and small learning rates that are often violated in practice.
• Look-ahead prediction is one-step and the first-order interpretation holds only under local risk drift.

Experimental setup

Datasets: Countdown-34 Countdown-4 few-shot sinusoid regression MuJoCo environments Ergo-Reacher Lunar-Lander

Benchmarks: Countdown-34 Countdown-4 HalfCheetahVel Walker2dVel ReacherPos LunarLander ErgoReacher few-shot sinusoid regression

Metrics: CVaR_alpha average return Spearman rank correlation test accuracy MSE

Baselines: ERM DRM GDRM OHTM MPTS PCL GRESO MoPPS

Builds on: model predictive task sampling MAML GRPO PDTS

Closest papers by meaning

• Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments (ICML 2025 · similarity 0.90)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.88)
• Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.88)
• Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.88)
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (NeurIPS 2025 · similarity 0.87)