Research map / Adversarial robustness and security / Attacks on text and graphs
Training Socially Aligned Language Models on Simulated Social Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SandBox, a simulated society of 100 LLM social agents, and the Stable Alignment framework (imitation, self-critic, realignment stages with a contrastive preference loss) that trains LLaMA-7B on the resulting interaction data without a reward model.
Problem
Language models trained on static corpora or with reward models generalize poorly to unfamiliar social scenarios and are vulnerable to jailbreaking, and reward-model-based RLHF suffers from misspecification and reward gaming.
Approach
Simulate a society of LM-based social agents on a grid that exchange feedback and revise responses under a latent rule, then train an LM directly on the recorded ratings, feedback, and revisions via three stages (imitation, self-critic, realignment) with a contrastive preference objective, removing the need for an online reward model.
Key results
- On HH-Adversarial, the full three-stage method reaches an alignment score of 8.23 versus 2.52 for Alpaca, 6.24 for RRHF and 8.43 for ChatGPT.
- The full method scores 0.53 MC1 on TruthfulQA versus 0.38 for RRHF and 0.60 for ChatGPT.
- Human annotators judged the method at least as good as ChatGPT 66% (HHH) and 69% (HHH-A) of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Stable Alignment (IL + SC + RA) instead of DPO | LLaMA 7B · HH | Alignment: 6.54 -> 7.35 | Helped |
| Use Stable Alignment (IL + SC + RA) instead of RRHF | LLaMA 7B · HH-A | Alignment: 6.24 -> 8.23 | Helped |
| Use Stable Alignment (IL + SC + RA) instead of RRHF | LLaMA 7B · Moral Stories | ACC: 0.74 -> 0.78 | Helped |
| Use Stable Alignment (IL + SC + RA) instead of RRHF | LLaMA 7B · MIC | ACC: 0.67 -> 0.73 | Helped |
| Use Stable Alignment (IL + SC + RA) instead of RRHF | LLaMA 7B · ETHICS-Deontology | ACC: 0.63 -> 0.65 | Helped |
| Use Stable Alignment (IL + SC + RA) instead of RRHF | LLaMA 7B · TruthfulQA | MC1: 0.38 -> 0.53 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · HH-A | Alignment: 6.59 -> 8.23 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · HH | Alignment: 6.56 -> 7.35 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · Moral Stories | ACC: 0.72 -> 0.78 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · MIC | ACC: 0.68 -> 0.73 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · ETHICS-Deontology | ACC: 0.64 -> 0.65 | Helped |
| Add Realignment (RA) to IL + SC (ablation) | Stable Alignment · TruthfulQA | MC1: 0.47 -> 0.53 | Helped |
Surprising findings
- Scaling from a 6.8B model to 175B GPT-3 did not significantly improve alignment in the simulated society.
- Instruction tuning on Alpaca data degraded adversarial robustness relative to base LLaMA (HH-A 2.52 vs 3.28).
Limitations
- Text-only interactions, without non-verbal cues.
- Assumes a static view of societal norms that does not capture evolving values.
- Experiments are conducted only in English.
Experimental setup
Datasets: Anthropic HH-RLHF questions (9,662 controversial questions for simulation) Moral Stories MIC ETHICS TruthfulQA Alpaca instruction data
Benchmarks: Anthropic HH HH-Adversarial Moral Stories MIC ETHICS-Deontology TruthfulQA
Metrics: human-rated alignment score (1-10) accuracy MC1 Elo win rate human Likert ratings
Baselines: LLaMA Alpaca Alpaca + SFT TRLX Chain-of-Hindsight DPO RRHF ChatGPT
Builds on: Alpaca RLHF Direct Preference Optimization RRHF Chain-of-Hindsight
Closest papers by meaning
- Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation (ICML 2024 · similarity 0.88)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.86)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.86)
- Fundamental Limitations of Alignment in Large Language Models (ICML 2024 · similarity 0.86)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.86)
- Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.