Research map / Large language models / Alignment and preferences
Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMI, an iterative self-supervised method that finetunes a pretrained LM to maximize a contrastive lower bound on conditional mutual information between constitutions (principles) and self-generated responses, without preference labels or demonstrations.
Problem
Aligning a pretrained language model to follow a set of natural-language behavioral principles (a constitution) typically requires preference labels, SFT demonstrations, or in-context examples, which are resource-intensive.
Approach
Treat constitution-following as increasing conditional mutual information between constitutions and responses given queries; optimize an InfoNCE-style two-sided contrastive cross-entropy loss over constitution-response log-probabilities in an expert-iteration loop, with early-iteration regularization against over-optimization.
Key results
- SAMI-trained mistral-7b reaches 66% (helpful) and 77% (harmless) win rates vs. the base model at iteration three, and 54%/57% vs. mistral-7b-instruct.
- mixtral-8x7b trained with weak (mistral-7b-instruct) principles reaches 65% win rate vs. mistral-7b-instruct on TL;DR summarization.
- llama3-70b reaches up to 60% (68% with CoT) win rate on trained principles and 63% (67% with CoT) on held-out principles vs. base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SAMI finetuning | Mistral-7B · HH-RLHF helpful queries | length-corrected win rate vs. Mistral-7B: 66% | Helped |
| Apply SAMI finetuning | Mistral-7B · HH-RLHF harmless queries | length-corrected win rate vs. Mistral-7B: 77% | Helped |
| Apply SAMI finetuning | Mistral-7B · HH-RLHF single-turn dialogue | win rate vs. Mistral-7B-Instruct: 54%; 57% | Helped |
| Apply SAMI finetuning with Mistral-7B-Instruct-written principles | Mistral-7B · TL;DR | win rate vs. Mistral-7B: 71% | Helped |
| Apply SAMI finetuning with Mistral-7B-Instruct-written principles | Mistral-7B · TL;DR | win rate vs. Mistral-7B-Instruct: 47% | Hurt |
| Apply SAMI finetuning with Mistral-7B-Instruct-written principles | Mixtral-8x7B · TL;DR | win rate vs. Mixtral-8x7B: 63% | Helped |
| Apply SAMI finetuning with Mistral-7B-Instruct-written principles | Mixtral-8x7B · TL;DR | win rate vs. Mistral-7B-Instruct: 65% | Helped |
| Apply SAMI finetuning with Claude-Opus-written principles | Mistral-7B · TL;DR | win rate vs. Mistral-7B: 70% | Helped |
| Apply SAMI finetuning with Claude-Opus-written principles | Mistral-7B · TL;DR | win rate vs. Mistral-7B-Instruct: 44% | Hurt |
| Apply SAMI finetuning with Claude-Opus-written principles | Mixtral-8x7B · TL;DR | win rate vs. Mixtral-8x7B: 62% | Helped |
| Apply SAMI finetuning with Claude-Opus-written principles | Mixtral-8x7B · TL;DR | win rate vs. Mistral-7B-Instruct: 68% | Helped |
| Add chain-of-thought reasoning (ablation) | SAMI-trained Llama3-70B · TL;DR train principles | win rate vs. Llama3-70B: 60% -> 68% | Helped |
Surprising findings
- A weak instruction-finetuned model (mistral-7b-instruct) can write principles sufficient to align a stronger base model (mixtral-8x7b).
- Win rates against the instruct model are initially below chance for mistral-7b but surpass chance after three SAMI iterations on dialogue.
Limitations
- Evaluated only on two domains (dialogue and summarization) with a small set of principles; no multi-turn or multi-principle evaluation.
- Requires regularization (early-iteration updates) to avoid over-optimization and gibberish; training too long causes forgetting.
- Length bias similar to DPO; summarization needed explicit conciseness principle to mitigate.
Experimental setup
Datasets: HH-RLHF helpful-base HH-RLHF harmless-base OpenAI TL;DR comparisons
Benchmarks: HH-RLHF helpful-base (250 queries) HH-RLHF harmless-base (250 queries) TL;DR comparisons (250 queries)
Metrics: conditional mutual information lower bound length-corrected win rate sequence length
Baselines: mistral-7b-instruct mistral-7b (base model) DPO KTO LIMA
Builds on: InfoNCE expert iteration constitutional AI CLIP DPO
Closest papers by meaning
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.89)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.88)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.88)
- Aligning Language Models with Demonstrated Feedback (ICLR 2025 · similarity 0.88)
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.88)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.