Research map / Large language models / Reasoning and chain of thought

W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes W2S-AlignTree, a plug-and-play inference-time alignment method that runs MCTS over a strong LLM's token chunks, scoring nodes with the log-likelihood ratio of a weak aligned/unaligned model pair, with an entropy-aware PUCT selection rule.

Problem

LLM outputs are often misaligned with human preferences; training-time alignment (RLHF/DPO) is costly and gives little fine-grained control at inference, and weak-to-strong supervision is not used for real-time guidance.

Approach

Combine MCTS with weak-to-strong generalization: the strong model proposes step-level chunks, a small aligned weak model's log-ratio against its unaligned reference serves as the proxy reward, backpropagation uses max return, EA-PUCT adds an entropy-scaled exploration bonus, and a final global re-ranking picks the output.

Key results

• On summarization with Llama3-8B, W2S-AlignTree raises the gold reward from 1.89 to 2.19 (reported as 15.9% relative gain).
• On controlled sentiment generation with Qwen2.5-7B, r_gold reaches 4.79 vs 4.36 for second-best CBS (about 10% improvement).
• On OASST1 instruction following with Qwen2.5-7B under oasst-rm-2 gold RM, score 1.33 vs 0.91 for BoN.

Surprising findings

• On Qwen2.5-7B-Instruct and some instruction-following settings, W2S-AlignTree is not best; CBS wins (e.g., 1.67 vs 1.51 on Qwen2.5-7B-Instruct under oasst-rm).

Limitations

• Theoretical justification relies on strong assumptions (power-law relation between weak and strong distributions) that may not hold in practice.
• Results are not uniformly best; CBS outperforms on several configurations.
• Search cost of MCTS at inference time is not quantified.

Experimental setup

Datasets: IMDB TL;DR OASST1

Benchmarks: IMDB TL;DR OASST1

Metrics: gold reward score r_gold

Baselines: Base decoding Best-of-N Chunk-level Beam Search (CBS) N-UCT RT-UCT RT-PUCT CMB MMB

Builds on: Weak-to-Strong Generalization DPO PUCT Chunk-level beam search MCTS-DPO

Closest papers by meaning

• Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.92)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
• InfAlign: Inference-aware language model alignment (ICML 2025 · similarity 0.90)
• Test-Time Alignment for Large Language Models via Textual Model Predictive Control (ICLR 2026 · similarity 0.89)
• Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction (AAAI 2025 · similarity 0.89)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.89)