Research map / Large language models / Alignment and preferences

Aligning Large Language Models with Representation Editing: A Control Perspective

NeurIPS 2024 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-Control aligns frozen LLMs at test time by training a value function on hidden states via a one-step Bellman policy-iteration target and performing gradient ascent on the logit-layer state to steer generation toward higher reward.

Problem

Aligning LLMs with human objectives usually requires costly, unstable fine-tuning, while existing test-time methods (prompting, guided decoding) leave the base model unchanged and are limited by its capabilities or slow.

Approach

View an autoregressive LM as a discrete-time stochastic dynamical system, add control signals to its hidden/logit state, train a value function on hidden states with a one-step policy iteration Bellman target, and at test time optimize the control by gradient ascent on the value score with small step size and few updates acting as implicit regularization.

Key results

• On HH-RLHF with Vicuna-7B, Re-Control + prompting reaches 80.3% GPT-4 win rate and average reward 6.267, versus 74.6% for the strongest baseline CD prefix (reward 6.105 for CD prefix) and 57.6% for base.
• Re-Control alone on Llama3-8B SHP achieves 71.0% win rate vs 56.3% base and 64.0% CD prefix, at 0.21 h inference versus 12.16 h for CD prefix.
• Compared with LoRA-based PPO and DPO fine-tuning on Vicuna-7B, Re-Control is described as a competitive alternative; the paper reports it is about 20x faster than controlled decoding at similar performance.

Measured effects reported in the paper

Surprising findings

• Controlled decoding combined with prompting shows only marginal gains, whereas Re-Control combined with prompting improves substantially.
• Performance of Re-Control first rises then falls with more test-time iterations, attributed to reward over-optimization, with coherence and diversity dropping to near zero.

Limitations

• Value function is trained only on the last-layer hidden state with a single-step policy iteration; multi-layer or more iterations are left to future work.
• Test-time optimization can lead to reward over-optimization, degrading coherence and diversity if too many updates are used.
• Only single-objective alignment with a fixed reward model is studied; multi-objective alignment is not explored.

Experimental setup

Datasets: HH-RLHF Stanford SHP HarmfulQA

Benchmarks: HH-RLHF Stanford SHP HarmfulQA

Metrics: GPT-4 win rate average reward diversity coherence inference time

Baselines: Prompt Engineering Controlled Decoding (ARGS) CD prefix Static Representation Editing PPO (LoRA) DPO (LoRA)

Builds on: controlled decoding representation engineering policy iteration DPO

Closest papers by meaning

• Test-Time Alignment for Large Language Models via Textual Model Predictive Control (ICLR 2026 · similarity 0.92)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
• Alignment Dynamics in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.90)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
• Progressively Label Enhancement for Large Language Model Alignment (ICML 2025 · similarity 0.90)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)