Research map / Large language models / AI and society
ELEPHANT: Measuring and understanding social sycophancy in LLMs
Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compare LLM responses to human responses on open-ended advice | 11 LLMs · OEQ | Validation rate: 22% (human) -> 72% (LLM) | Hurt |
| Compare LLM responses to human responses on indirectness | 11 LLMs · OEQ | Indirectness rate: 21% (human) -> 66% (LLM) | Hurt |
| Present both sides of a moral conflict to the model | 11 LLMs · AITA-NTA-FLIP | Moral sycophancy: 48% of cases say NTA to both sides | Hurt |
| Use preferred rather than dispreferred responses in preference datasets | LMSys, UltraFeedback, PRISM, HH-RLHF · Preference datasets | Validation and indirectness rate | Hurt |
| Preferred vs dispreferred responses on framing sycophancy | Preference datasets · Preference datasets | Framing rate | No clear effect |
| Prepend instruction to be less sycophantic when appropriate (ablation) | GPT-4o, Llama-70B · OEQ | Social sycophancy score: GPT-4o OEQ validation 0.71, framing -0.58 | Mixed |
| Rewrite prompts in the third person (ablation) | GPT-4o, Llama-8B, Llama-70B · OEQ, AITA, SS | Social sycophancy score: OEQ validation GPT-4o 0.71 -> 0.45 (instruction vs perspective row) | Mixed |
| Use truthfulness-tuned ITI model (ablation) | Llama-70B vs Llama-8B · OEQ | Validation sycophancy: 8B 0.56; 70B 0.18 | Mixed |
| Fine-tune with DPO against validation and indirectness (ablation) | Llama-8B · OEQ | Validation sycophancy: -0.12 (DPO-Validation) | Improved |
| Fine-tune with DPO against framing sycophancy (ablation) | Llama-8B · OEQ | Framing sycophancy | No clear effect |
| Increase model size within Llama or Mistral families | Llama and Mistral models · ELEPHANT | Social sycophancy | No clear effect |
Closest papers by meaning
- When Helpfulness Becomes Sycophancy: Sycophancy is a Boundary Failure Between Social Alignment and Epistemic Integrity in Large Language Models (NeurIPS 2026 · similarity 0.90)
- Position: Rethinking LLM Bias Probing Using Lessons from the Social Sciences (ICML 2025 · similarity 0.86)
- DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life (ICLR 2025 · similarity 0.86)
- Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior (NeurIPS 2026 · similarity 0.86)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.86)
- Towards Understanding Sycophancy in Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
