Research map / Large language models / Alignment and preferences
SPRI: Aligning Large Language Models with Context-Situated Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPRI, which has a base model generate and critique-refine principles tailored to each input query, then uses them to guide and refine responses, evaluated on reappraisal, BiGGen rubric generation, and SFT data synthesis.
Problem
Generic, static constitutional principles are hard to apply to nuanced, context-dependent alignment and evaluation, while expert-crafted or instance-specific principles and rubrics require substantial human effort.
Approach
Generate query-specific principles in real time with a base model, iteratively refined by a critic model using direct-assessment feedback, then use those principles both to guide the response and as rubrics for critique and refinement; the critic need not be stronger than the base model.
Key results
- On reappraisal, SPRI improves alignment by an average of 6.1% and empathy by 8.4% over the strongest vanilla baseline, without oracle principles.
- On BiGGen Bench rubric generation, SPRI outperforms the best instance-agnostic MT-Bench baseline by an average of 12.1% in Pearson correlation using only 3 seed rubrics.
- SFT on SPRI-generated data improves TruthfulQA over off-the-shelf models by an average of 24.76% and over oracle-response fine-tuning by 19.09%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use context-situated principles rather than a generic vanilla prompt | SPRI with GPT-4o-mini · Cognitive reappraisal | alignment: 7.90 -> 8.00 | Helped |
| use context-situated principles rather than a generic vanilla prompt | SPRI with Llama-3.1-70B-Instruct · Cognitive reappraisal | empathy: 4.43 -> 4.77 | Helped |
| generate instance-specific rubrics with SPRI rather than use MT-Bench rubric | BiGGen Bench evaluators · BiGGen Bench | Pearson correlation | Mixed |
| fine-tune on SPRI-generated data rather than oracle responses | Llama-3.1-8B · TruthfulQA | accuracy: 41.62% -> 55.92% | Helped |
| fine-tune on SPRI-generated data rather than oracle responses | Llama-3.1-8B-Instruct · TruthfulQA | accuracy: 46.75% -> 54.69% | Helped |
| fine-tune on SPRI-generated data rather than oracle responses | Mistral-7B-v0.3 · TruthfulQA | accuracy: 40.42% -> 51.85% | Helped |
| fine-tune on SPRI-generated data rather than oracle responses | Mistral-7B-v0.3-Instruct · TruthfulQA | accuracy: 42.87% -> 56.43% | Helped |
| fine-tune on SPRI-generated data rather than oracle responses | Gemma-2-9B · TruthfulQA | accuracy: 44.81% -> 55.72% | Helped |
| fine-tune on SPRI-generated data rather than oracle responses | Gemma-2-9B-it · TruthfulQA | accuracy: 47.11% -> 62.62% | Helped |
| add seed examples to principle generation (ablation) | SPRI · Cognitive reappraisal | alignment: 7.67 -> 8.00 | Helped |
| add seed examples to principle generation (ablation) | SPRI · BiGGen Bench | Pearson correlation: 0.410 -> 0.472 | Helped |
| use task-relevant seed examples instead of generic default-principle seeds (ablation) | SPRI · Cognitive reappraisal | alignment: 7.67 -> 8.00 | Helped |
Surprising findings
- Using default (irrelevant) principles alone sharply degrades reappraisal alignment, e.g. GPT-4o-mini drops to 2.13 from about 8.0, showing context-specific principles matter.
Limitations
- Reappraisal evaluation uses only 30 Reddit posts and automatic GPT-4 judging.
- Critic cost is substantial, mainly from the critic model calls, and the critic in experiments is Prometheus-2-8x7B.
- Reliance on LLM judges for evaluation rather than human judgments in SFT experiments.
Experimental setup
Datasets: Dolly MixInstruct BiGGen Bench Reddit emotional-support posts (RESORT data)
Benchmarks: RESORT reappraisal dataset BiGGen Bench TruthfulQA MUSR GPQA BBH MMLU-Pro Hellaswag
Metrics: average alignment score empathy score Pearson correlation with human labels TruthfulQA accuracy harmfulness factuality
Baselines: vanilla prompting self-refine oracle principles MT-Bench rubric FLASK rubric oracle rubrics self-instruct self-align topic-guided red-teaming seed principles
Builds on: Constitutional AI Self-Refine RESORT Prometheus
Closest papers by meaning
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.88)
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.88)
- Generative Judge for Evaluating Alignment (ICLR 2024 · similarity 0.87)
- PaCE: Parsimonious Concept Engineering for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.87)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.