Research map / Large language models / Reasoning and chain of thought

Consolidating Rewarded Perturbations for LLM Post-Training

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoRP, a gradient-free operator that consolidates a population of rewarded Gaussian weight perturbations into a single deployable model via reward-weighted, compatibility-aware aggregation gated by held-out validation.

Problem

RandOpt's weight-space random search for post-training needs K forward passes per test example via prediction-level ensembling and does not extend cleanly to free-form generation; the question is whether the rewarded perturbation population can be folded into one model.

Approach

Reward-weighted averaging of elite perturbations is refined by reweighting each perturbation by its alignment with the provisional mean and its orthogonal dispersion, then gated by a held-out validation/probe procedure that can abstain; a split-half analysis shows reproducible low-rank structure in all 25 model-task pairs.

Key results

• CoRP improves the base model by 8.1 points on average over 25 model-task pairs, scoring 50.0 vs. 41.9 for base and 56.3 for RandOpt K=50.
• With N=500 perturbations (one tenth of RandOpt's budget) and one forward pass, CoRP exceeds RandOpt K=1 by 6.5 points on average.
• The split-half subspace-excess statistic is stable (positive 95% lower bound) on 25 of 25 model-task pairs, while the mean-consensus statistic is stable on only 10.

Surprising findings

• Reward-weighted mean direction is reproducible in only 10 of 25 pairs, while low-rank subspace structure is reproducible in all 25.
• Sparse TIES-style merging of random rewarded perturbations underperforms the base model on three of four ablation cells.

Limitations

• Requires a reward or verifier on a support set, so it does not address settings without verifiable rewards.
• Evaluated only on tasks with categorical or easily verifiable outputs; free-form generation is left open.
• Gap to the K=50 ensemble remains, widest on smaller models.

Experimental setup

Datasets: Countdown GSM8K OlympiadBench ROCStories MBPP

Benchmarks: Countdown GSM8K OlympiadBench ROCStories MBPP

Metrics: test accuracy pass@1 average score over 25 model-task pairs gain composition (reasoning fix, format fix, regression)

Baselines: PPO GRPO RandOpt K=1 RandOpt K=50 reward-weighted averaging top-r projection TIES-style sparse merging

Builds on: RandOpt PPO GRPO TIES-merging covariance-adaptive black-box search

Closest papers by meaning

• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
• On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference (ICML 2026 · similarity 0.88)
• Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.88)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.88)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.88)