Research map / Large language models / Alignment and preferences
Fundamental Limitations of Alignment in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Behavior Expectation Bounds (BEB), a theoretical framework modeling an LLM as a mixture of ill- and well-behaved components, and proves that any behavior with nonzero probability can be elicited by an adversarial prompt whose length grows logarithmically in 1/alpha, with LLaMA-2 experiments supporting the mechanism.
Problem
Whether LLM alignment (RLHF and prompting) can be robust to adversarial prompting, i.e., whether undesired behaviors with small nonzero probability can be elicited by prompts.
Approach
Decompose the LLM distribution into a mixture of ill-behaved and well-behaved components with distinguishability (alpha, beta, gamma) assumptions; prompts reweight the mixture prior, so a sufficiently long prompt sampled from the negative component pushes the whole model into negative behavior.
Key results
- Estimated beta=20, beta'=30, sigma^2=50 for Llama-2 13B chat on agreeableness, giving sigma/beta=0.35 and beta'/beta=1.5.
- Ratio (1/beta)log(1/alpha) about 3 estimated for RLHF Llama-2 13B chat, matching the approximate misaligning prompt length of a few sentences.
- Pretrained models show beta about 5 times smaller than the RLHF model on both agreeableness and anti-immigration, suggesting RLHF may lower alpha while raising beta.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the length of prompts sampled from the negative-behavior proxy | RLHF fine-tuned LLaMA 2 13B chat · Agreeableness | Behavior expectation | Hurt |
| Use an aligning prefix prompt | RLHF fine-tuned LLaMA 2 13B chat · Agreeableness | Misalignment delay: about one sentence | Helped |
| Use a negative-behavior LLM proxy to construct adversarial prompts | RLHF fine-tuned LLaMA 2 13B chat · Agreeableness | Behavior expectation | Hurt |
| Apply RLHF alignment rather than use the pretrained model | LLaMA models · Agreeableness and anti-immigration | β distinguishability: 5 times smaller | Mixed |
| Add a negative behavior sentence after a neutral prefix | LLaMA 2 13B chat distribution comparison · Agreeableness | β prompt-distinguishability: β≈20 | No clear effect |
Surprising findings
- RLHF may reduce the probability of negative behavior (alpha) while increasing its distinguishability (beta), potentially making undesired behaviors easier to elicit via prompts.
Limitations
- Results are sentence-wise and rely on distinguishability and mixture assumptions that may not hold for real LLMs.
- Experiments use LoRA-finetuned proxies for the negative component rather than true subcomponents.
- Misaligning prompt construction is not computationally accessible in practice without a proxy; bounds are upper bounds.
Experimental setup
Datasets: behavior datasets of Perez et al. (agreeableness, anti-immigration statements)
Benchmarks: agreeableness behavior dataset anti-immigration behavior dataset
Metrics: KL divergence log-likelihood ratio variance behavior expectation
Builds on: PAC framework for in-context learning (Xie et al.) RLHF representation engineering
Closest papers by meaning
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Single Character Perturbations Break LLM Alignment (AAAI 2025 · similarity 0.89)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.88)
- Why Do Some Language Models Fake Alignment While Others Don't? (NeurIPS 2025 · similarity 0.88)
- Jailbreaking as a Reward Misspecification Problem (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.