Research map / Large language models / Alignment and preferences

Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Residual Alignment Model (RAM), which frames LLM alignment as importance sampling with a frozen upstream proposal model and a small autoregressive residual aligner, trained at sequence level and decoded token-by-token with a low first-token latency.

Problem

Alignment of large pretrained LLMs usually requires retraining the full model, and existing decoupled alignment modules (e.g., Aligner) suffer from high first-token latency and out-of-distribution inputs from reference responses.

Approach

Formalizes residual alignment as importance sampling: the product of a frozen proposal LM and a separately trained autoregressive residual aligner defines the aligned distribution. The aligner is trained by a sequence-level SFT-derived loss and decoded via a propose-align-reduce token-level self-normalized importance sampling scheme that avoids waiting for full upstream responses.

Key results

• On UltraChat, Llama3.1-8B proposal with a 1B residual aligner reaches 6.46 LC / 3.68 WR (Qwen2.5-Eval) vs 2.34 / 1.60 for the Aligner variant, and the 8B warmed-up model alone gets 5.06 / 2.93.
• On Anthropic-HH, DPO 8B with a 1B residual aligner improves GPT4-Eval helpfulness LC from 68.03 to 72.22, and DPO 14B with a 3B aligner improves it from 74.53 to 75.39 (WR 74.25 to 74.75).
• First-token latency for Llama8B+R.A. 1B is 0.31 s versus 10.14 s for the Aligner variant and 0.022 s for SFT alone.

Measured effects reported in the paper

Surprising findings

• The Aligner baseline underperforms even the unaligned SFT proposal model on Anthropic-HH, attributed to OOD reference-response inputs at inference.

Limitations

• Requires the proposal and residual aligner to share a vocabulary, limiting use across model families and closed-source models.
• Token-level sampling can add overhead beyond SFT in decoding (first-token latency 0.31 s vs 0.022 s for SFT).
• Performance gains from larger residual aligners are modest, and the method is evaluated only on two open model families.

Experimental setup

Datasets: UltraChat TL;DR Summarization Anthropic-HH

Benchmarks: AlpacaEval 2 TL;DR Summarization test split Anthropic-HH helpful-base and harmless-base test split

Metrics: length-controlled win rate win rate first-token latency

Baselines: Aligner SFT DPO Controlled Decoding warmed-up proposal model

Builds on: Aligner Residual EBM Controlled Decoding DPO

Closest papers by meaning

• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.91)
• Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.90)
• Learn Your Reference Model for Real Good Alignment (ICLR 2025 · similarity 0.90)
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.89)
• Progressively Label Enhancement for Large Language Model Alignment (ICML 2025 · similarity 0.89)
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.89)