Research map / Large language models / Instruction and fine-tuning
Alignment for Honesty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines honesty for LLMs as answering known questions and abstaining ("idk") on unknown ones, proposes prudence/over-conservativeness/honesty metrics and four honesty-oriented SFT variants (Absolute, Confidence, Multisample) on LLaMA2-Chat using TriviaQA.
Problem
LLMs hallucinate confidently answers to questions beyond their knowledge; aligning them for honesty requires knowing when to refuse, without becoming over-conservative, and needs metrics and training methods to do so.
Approach
Approximate a model's knowledge boundary by sampling its answers to each question (expected accuracy), then fine-tune the model to answer known questions and respond with idk phrases to unknown ones, with evolutionary metrics (prudence, over-conservativeness, honesty score) comparing before and after alignment.
Key results
- On TriviaQA with LLaMA2-Chat-13B, Multisample reaches honesty score 75.91 (vs 50.00 unaligned) with prudence 67.72 but over-conservativeness 15.89.
- Confidence-Verb keeps accuracy at 73.34 on TriviaQA, close to unaligned 73.71, and on Non-AmbigQA raises honesty to 68.74 from 50.06.
- Helpfulness on Eval-P- stays near unaligned: GPT-4 score 8.61 for Confidence-Verb vs 8.62 unaligned.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply honesty-oriented supervised fine-tuning (Absolute) rather than prompting alone | LLaMA2-Chat-13B · TriviaQA | honesty: 60.64 -> 68.88 | Helped |
| Apply honesty-oriented supervised fine-tuning (Multisample) rather than prompting alone | LLaMA2-Chat-13B · TriviaQA | honesty: 60.64 -> 75.91 | Helped |
| Use expected accuracy as a training signal with numerical confidence expressions rather than Absolute (ablation) | LLaMA2-Chat-13B · TriviaQA | honesty: 68.88 -> 74.37 | Helped |
| Use expected accuracy as a training signal with verbal confidence expressions rather than Absolute (ablation) | LLaMA2-Chat-13B · TriviaQA | honesty: 68.88 -> 74.12 | Helped |
| Train on multiple sampled responses per question rather than Absolute (ablation) | LLaMA2-Chat-13B · TriviaQA | honesty: 68.88 -> 75.91 | Helped |
| Train on multiple sampled responses per question rather than Absolute (ablation) | LLaMA2-Chat-13B · TriviaQA | accuracy: 71.30 -> 68.88 | Hurt |
| Use honesty-oriented fine-tuning (Multisample) rather than prompting alone | LLaMA2-Chat-13B · Non-AmbigQA | honesty: 57.39 -> 70.18 | Helped |
| Use honesty-oriented fine-tuning (Confidence-Verb) rather than the unaligned model | LLaMA2-Chat-13B · Non-AmbigQA | accuracy: 49.63 -> 49.54 | No clear effect |
| Use confidence-oriented fine-tuning (Confidence-Num) rather than prompting alone | LLaMA2-Chat-13B · PUQA | prudence: 28.90 -> 87.30 | Helped |
| Use honesty-oriented fine-tuning (Confidence-Verb) rather than the unaligned model | LLaMA2-Chat-13B · PKQA | accuracy: 100.00 -> 96.80 | Hurt |
| Fine-tune on gold answers for unknown questions instead of idk responses | LLaMA2-Chat-13B · PKQA | accuracy: 100.00 -> 87.70 | Hurt |
| Align with honesty using Confidence-Verb rather than leaving the model unaligned | LLaMA2-Chat-13B · Eval-P⁻ | helpfulness (GPT-4): 8.62 -> 8.61 | No clear effect |
Surprising findings
- Plain fine-tuning on gold answers for unknown questions (Fine-tuned baseline) drops PKQA accuracy by about 10 points, suggesting it teaches hallucination.
Limitations
- Knowledge boundary approximated by external behavior (answer correctness), not internal knowledge.
- Over-conservativeness rises for Multisample (15.89), trading some known-question answers for prudence.
- Correctness judged via string match plus ChatGPT, which may be imperfect.
Experimental setup
Datasets: TriviaQA Non-AmbigQA PUQA PKQA MMLU Eval-P
Benchmarks: TriviaQA Non-AmbigQA PUQA PKQA MMLU Eval-P
Metrics: prudence score over-conservativeness score honesty score accuracy
Baselines: Unaligned LLaMA2-Chat-13B Fine-tuned baseline (gold answers for unknown questions) Prompt-based idk Absolute Confidence-Num Confidence-Verb
Builds on: LLaMA2-Chat TriviaQA Eval-P Auto-J
Closest papers by meaning
- HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.92)
- Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.91)
- MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions (AAAI 2026 · similarity 0.89)
- How do Large Language Models Navigate Conflicts between Honesty and Helpfulness? (ICML 2024 · similarity 0.89)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.89)
- DENEVIL: TOWARDS DECIPHERING AND NAVIGATING THE ETHICAL VALUES OF LARGE LANGUAGE MODELS VIA INSTRUCTION LEARNING (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.