Research map / Large language models / Alignment and preferences
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MATRIX, a multi-role LLM social-scene simulator (Monopolylogue) whose consequence summaries drive self-critique and revision, then distills the results into the LLM via SFT for self-alignment, with a theoretical comparison to Constitutional AI.
Problem
Aligning LLMs with human values without external supervision (human annotation or stronger LLMs) and without rigid predefined rules that do not adapt to diverse user instructions.
Approach
A single LLM plays multiple social roles (user agent, reactive agents, objects) plus a social modulator that checks feasibility and distributes messages; the simulated social consequences are summarized and used for instruction-specific self-critique and revision, and the resulting data is distilled via SFT so inference speed is unchanged.
Key results
- The LLM with MATRIX beats the Vanilla 30B model with 81-91% win rates and beats ChatGPT (GPT-3.5-Turbo) on all four datasets in the 30B pairwise table.
- The MATRIX-tuned 30B LLM retains general ability, scoring 8.49 on Vicuna-Bench (vs 8.37 vanilla) and 6.99 on MT-Bench (equal to vanilla).
- Human ratings (875 in the abstract; 1750 total across 13B and 30B in the text) show the tuned 13B model exceeds GPT-4 in value alignment on PKU-SafeRLHF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MATRIX-generated social consequences at inference | Wizard-Vicuna-Uncensored 30B · HH-RLHF, Safe-RLHF, AdvBench, HarmfulQA | win rate vs. ChatGPT (GPT-3.5-Turbo): 57%; 71%; 91%; 54% | Helped |
| Fine-tune on MATRIX-generated data | Wizard-Vicuna-Uncensored 30B · HH-RLHF, Safe-RLHF, AdvBench, HarmfulQA | win rate vs. ChatGPT (GPT-3.5-Turbo): 65%; 64%; 74%; 58% | Helped |
| Add harmless data to training (ablation) | MATRIX-tuned LLM · HH-RLHF | win rate vs. base model: 89.3% | Helped |
| Add helpful data to harmless-data training (ablation) | MATRIX-tuned LLM · HH-RLHF | win rate vs. base model: 89.3% -> 85.2% | Hurt |
| Add helpful data to harmless-data training (ablation) | MATRIX-tuned LLM · MT-Bench | score: 6.50 -> 6.94 | Helped |
| Add simulation data to harmless and helpful training data (ablation) | MATRIX-tuned LLM · HH-RLHF | win rate vs. base model: 85.2% -> 93.3% | Helped |
| Add simulation data to harmless and helpful training data (ablation) | MATRIX-tuned LLM · MT-Bench | score: 6.94 -> 6.99 | Helped |
| Add simulation data to harmless and helpful training data (ablation) | MATRIX-tuned LLM · HH-RLHF | further MATRIX win rate: 63.3% -> 95.6% | Helped |
| Increase the number of agents in the simulation (ablation) | MATRIX · Value-alignment evaluation | harmless response ratio | Helped |
| Increase the number of interactions in the simulation (ablation) | MATRIX · Value-alignment evaluation | harmless response ratio | Helped |
| Increase the base model size from 7B to 30B (ablation) | MATRIX-tuned LLM · GPT-3.5-Turbo comparison | win rate | Helped |
| Apply MATRIX again to the tuned model (ablation) | MATRIX-tuned LLM · HH-RLHF | win rate vs. base model: 93.3% -> 95.6% | Helped |
Surprising findings
- Human evaluation reportedly shows a tuned 13B model outperforming GPT-4 on value alignment, while GPT-4 judges rank it below GPT-4 in some settings.
- Reported RLcd and RLAIF comparisons show the MATRIX-tuned LLM losing or tying on some datasets, e.g., RLcd wins only 44% on AdvBench.
Limitations
- Only explores value alignment (harmlessness); general tool-using roles not yet tested.
- Simulation at inference is time-consuming, motivating SFT distillation.
- Theoretical assumptions (collective advantage, stable critique generation, alignment chance) are argued as plausible rather than verified.
Experimental setup
Datasets: HH-RLHF PKU-SafeRLHF AdvBench HarmfulQA Vicuna-Bench MT-Bench
Benchmarks: HH-RLHF PKU-SafeRLHF AdvBench HarmfulQA Vicuna-Bench MT-Bench
Metrics: win rate tie rate lose rate human preference ratings MT-Bench score inference time
Baselines: Self-Alignment Context Distillation Critique-Revise RAIN LLM Debate Best-of-N Sampling Mistake Analysis Stable Alignment RLcd RLAIF ChatGPT (GPT-3.5-Turbo) GPT-4
Builds on: Constitutional AI Self-Align LLM Debate QLoRA
Closest papers by meaning
- Training Socially Aligned Language Models on Simulated Social Interactions (ICLR 2024 · similarity 0.88)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.88)
- Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.87)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.86)
- SPRI: Aligning Large Language Models with Context-Situated Principles (ICML 2025 · similarity 0.86)
- Human Simulacra: Benchmarking the Personification of Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.