Research map / Large language models / Alignment and preferences
Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COUPLE, a structural-causal-model plus counterfactual reasoning framework (value attribution, intervention, counterfactual prediction) that steers LLM responses to fine-grained pluralistic value priority profiles, with reasoning-based SFT for small models.
Problem
Aligning LLMs with pluralistic human values expressed as multi-dimensional value priorities is hard because existing methods treat values as independent (value complexity) and cannot precisely steer nuanced or underrepresented priority profiles (value steerability).
Approach
Model value-to-behavior relations with an SCM over value dimensions, value concepts and responses, then apply abduction-intervention-prediction counterfactual reasoning via LLM prompting to generate responses for any target value profile; the same pipeline synthesizes data for SFT of smaller models.
Key results
- On DailyDilemma with GPT-4.1-mini, COUPLE reduces MAE to 0.355 and raises correlation to 0.848, versus 0.505 / 0.611 for the best prompt baseline Value Prompt.
- On Touché23-ValueEval with LLaMA3.1-8B, Reasoning SFT reaches MAE 2.039 and correlation 0.578, outperforming VIM (2.188 / 0.383) and CultureBank (2.372 / 0.408).
- With DeepSeek-R1 on Touché23-ValueEval, COUPLE attains MAE 1.082 and correlation 0.798 against 1.720 / 0.708 for Value Prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COUPLE rather than Tree of Thought | GPT-4.1-mini · Touché23-ValueEval | MAE: 1.975 -> 1.433 | Helped |
| Use COUPLE rather than Plan and Solve | GPT-4.1-mini · DailyDilemma | Correlation: 0.632 -> 0.848 | Helped |
| Use reasoning-based rather than naive SFT | Open-source models · Touché23-ValueEval | MAE: 2.091 -> 2.039; 2.188 -> 1.971 | Helped |
| Add structural value abduction (SCM) (ablation) | COUPLE · Touché23-ValueEval | MAE: 1.873 -> 1.433 | Helped |
| Add structural value abduction (SCM) (ablation) | COUPLE · DailyDilemma | Correlation: 0.548 -> 0.848 | Helped |
| Add key value concept extraction (ablation) | COUPLE · Touché23-ValueEval | MAE: 1.812 -> 1.433 | Helped |
| Add key value concept extraction (ablation) | COUPLE · DailyDilemma | Correlation: 0.888 -> 0.848 | No clear effect |
| Add counterfactual prediction (ablation) | COUPLE · Touché23-ValueEval | MAE: 1.546 -> 1.433 | Helped |
| Add counterfactual prediction (ablation) | COUPLE · Touché23-ValueEval | Correlation: 0.779 -> 0.778 | Hurt |
| Add counterfactual prediction (ablation) | COUPLE · DailyDilemma | MAE: 0.381 -> 0.355 | Helped |
| Add SCM and counterfactual reasoning (ablation) | COUPLE · Touché23-ValueEval | MAE: 2.182 -> 1.433 | Helped |
Surprising findings
- For Qwen2.5-7B on Touché23-ValueEval, COUPLE prompting (MAE 2.547, correlation 0.408) is worse than Plan and Solve (2.129, 0.513), so prompt-based gains are limited on small open models.
Limitations
- Requires strong reasoning capability from the underlying LLM, so it is less effective on smaller models for inference-time alignment.
- Represents value objectives only as 5-level priorities over a limited set of value dimensions, which may not capture the richness of human values.
Experimental setup
Datasets: Touché23-ValueEval DailyDilemma European Social Survey (ESS) value profiles
Benchmarks: Touché23-ValueEval DailyDilemma
Metrics: MAE Spearman correlation human preference win/tie rate
Baselines: Raw Model Role Prompt Value Prompt Tree of Thought Plan and Solve CultureLLM CulturePark CultureSPA CultureBank VIM
Builds on: Schwartz Theory of Basic Values Plan-and-Solve prompting Tree of Thoughts CLADDER
Closest papers by meaning
- Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.87)
- Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models (NeurIPS 2026 · similarity 0.86)
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.85)
- DENEVIL: TOWARDS DECIPHERING AND NAVIGATING THE ETHICAL VALUES OF LARGE LANGUAGE MODELS VIA INSTRUCTION LEARNING (ICLR 2024 · similarity 0.84)
- Pairwise Calibrated Rewards for Pluralistic Alignment (NeurIPS 2025 · similarity 0.84)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.