Research map / Large language models / Instruction and fine-tuning
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes supervised pinpoint tuning (SPT), which identifies the ~4% of attention heads most responsible for sycophancy via path patching and fine-tunes only those heads, reducing sycophancy in Llama-2 and Mistral models with less general-capability loss than full SFT.
Problem
LLMs wrongly admit mistakes and abandon correct answers when users challenge them (sycophancy), and standard SFT fixes this at the cost of general capability.
Approach
Use path patching to locate a sparse set of sycophancy-related attention heads, verify them by knockout, then fine-tune only those heads' input/output projection matrices while freezing the rest of the network, including MLPs and embeddings.
Key results
- SPT on Llama-2-13B raises truthfulness from 18.89% to 86.72% and confidence from 0.08% to 71.92%, outperforming SFT (84.06%, 61.55%).
- SPT on Llama-2-13B improves GSM8K from 33.89% to 35.48%, while SFT drops it to 25.32%.
- SPT's KL divergence on Llama-2-13B is 0.0026 versus 0.0476 for SFT, about 20x smaller.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune pinpointed attention heads instead of full supervised fine-tuning | Llama-2-13B · SycophancyEval | confidence: 61.55 -> 71.92 | Helped |
| Tune pinpointed attention heads instead of full supervised fine-tuning | Llama-2-13B · SycophancyEval | truthfulness: 84.06 -> 86.72 | Helped |
| Tune pinpointed attention heads instead of full supervised fine-tuning | Llama-2-13B · GSM8K | accuracy: 25.32 -> 35.48 | Helped |
| Tune pinpointed attention heads instead of full supervised fine-tuning | Mistral-7B · SycophancyEval | confidence: 52.46 -> 69.68 | Helped |
| Increase the selected head count from 8 to 64 (ablation) | SPT on Llama-2-13B · SycophancyEval | confidence: 23.84 -> 71.92 | Helped |
| Select 64 heads randomly rather than pinpointing sycophancy-related heads (ablation) | SPT on Llama-2-13B · SycophancyEval | confidence: 71.92 -> 60.11 ±7.37 | Hurt |
| Add the top-1 MLP to the 64 tuned heads (ablation) | SPT on Llama-2-13B · SycophancyEval | confidence: 71.92 -> 75.82 | Helped |
| Add the top-1 MLP to the 64 tuned heads (ablation) | SPT on Llama-2-13B · SycophancyEval | accuracy before challenge: 46.99 -> 43.58 | Hurt |
| Add the top-1 MLP to the 64 tuned heads (ablation) | SPT on Llama-2-13B · SycophancyEval | accuracy after challenge: 47.55 -> 43.86 | Hurt |
| Tune pinpointed heads plus LoRA instead of SPT alone (ablation) | Llama-2-13B · SycophancyEval | confidence: 71.92 -> 86.33 | Helped |
| Use few-shot prompting instead of tuning | Llama-2-13B · SycophancyEval | confidence | No clear effect |
Surprising findings
- Only about 4% of attention heads have a noticeable direct effect on sycophantic output, and randomly chosen 64 heads give unstable results.
- Tuning an MLP alongside heads raised confidence but reduced accuracy before and after challenge, indicating more impairment than head tuning alone.
Limitations
- Path patching treats attention heads and MLPs as nodes; neuron-level units are not explored.
- Sycophancy evaluation follows a single definition and format; generalization to other formats is unclear.
- Judging relies on Qwen-72B-Chat, checked manually on only 100 samples.
Experimental setup
Datasets: SycophancyEval (MMLU, MATH, AQuA, TruthfulQA, TriviaQA) MMLU auxiliary train MATH train AQuA train TriviaQA train OpenWebText subset Sycophancy Evaluation Datasets (NLP, PHIL, POLI)
Benchmarks: SycophancyEval StrategyQA GSM8K HumanEval CSQA MMLU Sycophancy Evaluation Datasets (NLP, PHIL, POLI)
Metrics: confidence (answer confidence) truthfulness accuracy KL divergence apologize rate
Baselines: SFT LoRA DARE LoRA&SPT few-shot prompting
Builds on: path patching supervised fine-tuning LoRA DARE
Closest papers by meaning
- Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.90)
- Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.89)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.89)
- Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.