Research map / Large language models / Alignment and preferences
Constrain Alignment with Sparse Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FPO, a reference-based preference optimization method that replaces token-level sequential KL with a sparse-autoencoder feature-level MSE constraint and precomputes reference quantities offline, evaluated on Gemma-2-2B/9B with AlpacaEval-2, Arena-Hard and MT-Bench.
Problem
Preference alignment of LLMs with DPO-style methods is computationally expensive (token-level sequential KL over large vocabularies requires reference model and large storage) or unstable when reference-free (SimPO), leaving a gap between efficiency and controllability.
Approach
Combine SimPO-style length-normalized log-probability differences with a reference-based adaptive margin, and replace the TDPO sequential KL constraint with an MSE over top-k pooled sparse autoencoder feature activations, precomputing reference margins and activations offline.
Key results
- On Gemma-2-2B, FPO reports up to 5.08% win-rate improvement over other methods on AlpacaEval-2/Arena-Hard comparisons.
- FPO reaches 64.1% training accuracy and 1.68 entropy on UltraFeedback, second only to TDPO-2 in accuracy and tied for highest diversity.
- FPO reduces memory roughly 17% compared with TDPO while remaining close to SimPO in memory use.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FPO instead of the listed preference-optimization baselines | Gemma-2-2B and Gemma-2-9B · AlpacaEval-2, Arena-Hard, and MT-Bench | WR-L, WR, and Δ Score: FPO-vs-baseline values range from 49.5 to 55.1 WR and 0.0 to +0.5 Δ Score | Mixed |
| Replace TDPO-2 with FPO | Gemma-2-2B · UltraFeedback | Acc (%): 64.2 -> 64.1 | Hurt |
| Replace TDPO-2 with FPO | Gemma-2-2B · AlpacaEval2 entropy | H: 1.68 -> 1.68 | No clear effect |
| Use FPO instead of SimPO+KL | Gemma-2-2B · AlpacaEval-2 | WR (%): 50.8 -> 51.8 | Helped |
| Use FPO instead of SimPO+KL | Gemma-2-2B · AlpacaEval-2 | WR-L (%): 50.6 -> 50.2 | Hurt |
| Use a single SAE at layer 25 rather than multiple SAE layers (ablation) | FPO, Gemma-2-2B · UltraFeedback | Acc (%): 62.1 -> 64.1; 61.9 -> 64.1; 58.2 -> 64.1; 51.4 -> 64.1 | Helped |
| Use a single SAE at layer 25 rather than multiple SAE layers (ablation) | FPO, Gemma-2-2B · AlpacaEval-2 | WR (%): 47.2 -> 51.8; 48.4 -> 51.8; 48.6 -> 51.8; 47.8 -> 51.8 | Helped |
| Move SAE insertion from earlier layers to layer 25 (ablation) | FPO, Gemma-2-2B · UltraFeedback | Acc (%): 57.2 -> 63.6; 59.1 -> 63.6; 59.7 -> 63.6 | Helped |
| Use MLP SAE instead of residual SAE at tested layers (ablation) | FPO, Gemma-2-2B · UltraFeedback | Acc (%): 57.2 -> 57.4; 59.1 -> 61.3; 59.7 -> 62.4; 63.6 -> 63.4 | Mixed |
| Increase α from 0.1 to 2 (ablation) | FPO, Gemma-2-2B · UltraFeedback | Acc (%): 64.1 -> 61.9 | Hurt |
| Set α to 0.5 rather than 0.1 (ablation) | FPO, Gemma-2-2B · UltraFeedback | H: 1.630 -> 1.680 | Helped |
| Apply the stop-gradient operator (ablation) | FPO, Gemma-2-2B · UltraFeedback | H: 1.666 -> 1.652; 1.680 -> 1.680; 1.682 -> 1.682; 1.679 -> 1.679 | Mixed |
Surprising findings
- Adding multiple SAE layers did not consistently help and could degrade alignment metrics, with the single layer 25 setting performing best.
- Gains on Gemma-2-9B are smaller than on the 2B model, which the authors attribute to feature decomposition quality differences.
Limitations
- Gains on the 9B model are limited relative to the 2B model.
- Relies on pre-trained Gemma Scope SAEs; the theoretical KL bound depends on strong assumptions such as small logit differences and accurate reconstruction.
Experimental setup
Datasets: UltraChat-200K UltraFeedback AlpacaEval 2 Arena-Hard MT-Bench
Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench
Metrics: length-controlled win rate (WR-L) win rate (WR) MT-Bench score delta training accuracy entropy diversity
Baselines: SFT DPO TDPO-1 TDPO-2 SimPO SimPO+KL
Builds on: DPO SimPO TDPO Gemma Scope SAEs
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.90)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.90)
- Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.