Research map / Large language models / Alignment and preferences
Inverse Constitutional AI: Compressing Preferences into Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Inverse Constitutional AI, which compresses pairwise preference datasets into a short list of natural-language principles that an LLM annotator can use to reconstruct the original annotations.
Problem
Pairwise text preference datasets are used for training and evaluating LLMs, but they are hard to interpret and may contain unintended annotator biases that black-box reward models and aggregate statistics do not reveal.
Approach
Treat interpretation of preference data as compression: generate candidate principles per comparison with an LLM, cluster and subsample them, test each principle's votes against the original labels, and keep the top principles as a constitution whose reconstruction accuracy validates it.
Key results
- On the AlpacaEval unaligned split, GPT-4o constitutional annotators reach 61% agreement versus 34% for the default annotator at the full 648-pair scale (66% vs 27% at 130 pairs).
- On synthetic unaligned data, constitutional annotators improve agreement over the default annotator, while the ablation without filtering and testing fails to beat random in unaligned settings.
- Bias detection on 13k preferences finds verbosity bias (overly lengthy principle at 57.7% accuracy on PRISM and 57.1% on Chatbot Arena) and a numbered-list style bias (73.4% accuracy on AlpacaEval).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generated constitutions instead of the default annotator | GPT-4o annotator · AlpacaEval unaligned, full 648 preferences | agreement: 34% -> 61% | Helped |
| Use a personal constitution generated from that user's preferences | Chatbot Arena annotator · Individual users' Chatbot Arena annotations | reconstruction performance | Helped |
| Transfer a user's constitution to another user's annotations | Chatbot Arena annotator · Other user's Chatbot Arena annotations | reconstruction performance | Hurt |
| Generate a constitution from the target group's data | PRISM constitutional annotator · PRISM Group A and Group B preferences | reconstruction performance | Helped |
| Transfer a group's constitution to the other group's annotations | PRISM constitutional annotator · PRISM cross-group annotations | reconstruction performance | Hurt |
| Generate only one principle with a neutral prompt instead of diverse principles (ablation) | ICAI · Synthetic aligned dataset | agreement | Hurt |
| Remove deduplication (ablation) | ICAI · Synthetic aligned and synthetic unaligned datasets | agreement | Hurt |
| Remove deduplication (ablation) | ICAI · Synthetic orthogonal and AlpacaEval unaligned datasets | agreement | Helped |
| Remove principle testing and filtering (ablation) | ICAI · Synthetic orthogonal, synthetic aligned, synthetic unaligned, and AlpacaEval unaligned | agreement | Hurt |
| Use a GPT-4o-generated constitution rather than default annotation | GPT-4o constitutional annotator · AlpacaEval unaligned dataset | agreement | Helped |
| Use a GPT-4o-generated constitution rather than default annotation | GPT-3.5-Turbo constitutional annotator · AlpacaEval unaligned dataset | agreement | No clear effect |
Surprising findings
- GPT-3.5-Turbo fails to follow the GPT-4o-generated opposing constitution on the unaligned AlpacaEval split, revealing a capability gap between models.
- Removing deduplication improves results on the synthetic orthogonal and AlpacaEval unaligned datasets, suggesting repetition can help overcome strong model biases.
Limitations
- Non-causal: constitutions correlate with annotations but cannot show the original annotators used those principles (Rashomon effect, non-unique constitutions).
- Constitutions are a lossy compression of a few principles and may not reconstruct complex annotations.
- Preferences closely aligned with the LLM's priors are hard to improve upon, so the reconstruction loss is less informative in that case.
Experimental setup
Datasets: AlpacaEval Chatbot Arena Conversations Chatbot Arena Kaggle PRISM Anthropic HH-RLHF synthetic preference data
Benchmarks: AlpacaEval Chatbot Arena Conversations PRISM synthetic preference data
Metrics: annotation reconstruction accuracy relevance
Baselines: Default AlpacaEval annotator flipped Default annotator pretrained reward model fine-tuned reward model PopAlign
Builds on: Constitutional AI AlpacaEval PopAlign rule-based rewards
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- Anyprefer: An Agentic Framework for Preference Data Synthesis (ICLR 2025 · similarity 0.88)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.88)
- Same Words, Different Judgments: How Preferences Vary Across Modalities (NeurIPS 2026 · similarity 0.88)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.