Research map / Large language models / Instruction and fine-tuning

Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Gradient-free adaptation and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes weak-to-strong search, a chunk-level beam search that steers a frozen large LM at test time using the log-probability difference between a small tuned and untuned model pair.

Problem

Aligning large language models is resource-intensive via fine-tuning, while existing search-based test-time alignment methods either treat decoding as a bandit problem or need a value function trained from scratch due to sparse preference rewards.

Approach

Parametrize the sparse KL-constrained preference reward with the log-probability ratio of a small tuned vs. untuned model, which serves as both dense reward and value; then run chunk-level beam search (CBS) that samples chunks from the frozen large model and prunes by this score, enabling black-box and cross-vocabulary steering.

Key results

• On AlpacaEval 2.0 with Zephyr guidance, weak-to-strong search raises Llama-3-70B-Instruct LC win rate from 34.4% to 37.9% (BoN with N=4 reaches 36.6%).
• With Tulu guidance, Llama-3-70B-Instruct LC win rate reaches 39.1% and gpt-3.5-turbo-instruct rises from 16.0% to 19.8%.
• Weak-to-strong search with gpt2 guidance outperforms other test-time methods on summarization, nearly matching direct fine-tuning for most large models.

Measured effects reported in the paper

Surprising findings

• Vanilla beam search without external guidance does not consistently improve over regular decoding for instruction following.
• Weak guidance from a small model (zephyr-7b-beta, ~13% LC WR) improves stronger models (Llama-3-70B-Instruct at 34.4%).

Limitations

• EFT is inapplicable across different vocabularies, and CBS hyperparameters were not tuned.
• Instruction-following experiments depend on gpt-4-turbo as judge and on reusing existing small model pairs that cannot be reproduced exactly.
• Authors note high steerability can cause large KL deviation and over-optimization.

Experimental setup

Datasets: IMDB summarize_from_feedback AlpacaEval 2.0 prompts

Benchmarks: AlpacaEval 2.0 IMDB controlled-sentiment generation summarize_from_feedback

Metrics: gold reward length-controlled win rate raw win rate UltraRM score Starling-RM score

Baselines: Base decoding Best-of-N sampling Emulated Fine-Tuning Direct fine-tuning Vanilla beam search

Builds on: DPO emulated fine-tuning best-of-n sampling token-level MDP alignment

Closest papers by meaning

• W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search (AAAI 2026 · similarity 0.92)
• Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model (ICLR 2025 · similarity 0.92)
• Bayesian WeakS-to-Strong from Text Classification to Generation (ICLR 2025 · similarity 0.91)
• Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.90)
• A transfer learning framework for weak to strong generalization (ICLR 2025 · similarity 0.90)
• Prediction-Powered Ranking of Large Language Models (NeurIPS 2024 · similarity 0.89)