Research map / Large language models / Reasoning and chain of thought

Test-Time Alignment for Large Language Models via Textual Model Predictive Control

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Textual Model Predictive Control (TMPC), a test-time alignment framework that iteratively regenerates LLM responses conditioned on hindsight-identified high-reward subgoal segments stored in a buffer, evaluated on WMT'24 translation, HH-RLHF long-form responses, and MBPP.

Problem

Test-time alignment of frozen LLMs faces a trade-off: token-level guided decoding suffers from the curse of horizon (unreliable credit assignment), while response-level iterative refinement suffers from the curse of dimensionality (huge search space). Standard MPC also assumes hard segment boundarie

Approach

Recasts test-time alignment as receding-horizon trajectory optimization over text. Hindsight Subgoal Identification retrospectively extracts high-reward segments from rollouts into a buffer, and Subgoal-Conditioned Re-Generation uses buffered subgoals to condition the next round of full-response generation, with no fine-tuning of the LLM.

Key results

• On WMT'24 zh→en, TMPC reaches SEGALE_comet 94.62 with NA ratio 0.00, above Best-of-60 (90.97) and GPT-4o (94.58).
• On MBPP program synthesis, TMPC achieves a 61% pass rate, outperforming Best-of-35 and TPO (48% after 4 iterations).
• On HH-RLHF long-form responses, TMPC (3 iterations, 3 rollouts) beats DPO and Best-of-20 in GPT-4 head-to-head win rate while using about half the generations of Best-of-20.

Surprising findings

• Token-level and response-level test-time baselines such as ARGS and RAIN degrade translation badly (e.g., ARGS 63.99 vs. base 84.36 on zh→en), with high NA ratios.

Limitations

• Constrained by the expressiveness of the underlying LLM; cannot easily reach outputs far from the model's distribution.
• Relies on reward model quality; baseline comparisons depend on reward models trained by the authors.
• Per-response latency is high (108 s) relative to some baselines like RE-Control and GenARM.

Experimental setup

Datasets: WMT'24 Discourse-Level Literary Translation Dahoas/full-hh-rlhf MBPP

Benchmarks: WMT'24 Discourse-Level Literary Translation HH-RLHF MBPP

Metrics: SEGALE_comet NA ratio average reward GPT-4 win rate pass rate

Baselines: ARGS RAIN RE-Control GenARM TPO Best-of-N DPO SimPO SFT

Builds on: model predictive control MPPI TPO hierarchical reinforcement learning

Closest papers by meaning

• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.92)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
• W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search (AAAI 2026 · similarity 0.89)
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.89)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.89)