Research map / Large language models / Alignment and preferences

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

ICML 2024 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses the signaling bandits paradigm and a Gricean/Rational Speech Acts model to measure how LLMs trade off honesty (Quality) against helpfulness (decision-theoretic Relevance), and how RLHF, chain-of-thought, and prompting shift that trade-off.

Problem

LLMs must balance honesty and helpfulness, which conflict in everyday communication (e.g., rounding), yet benchmarks measure them separately; it is unclear how LLMs weigh them and how training and prompting shift that balance.

Approach

Formalizes honesty as the Gricean maxim of Quality and helpfulness as decision-theoretic utility under the Rational Speech Acts framework, combines them in a convex utility with weight lambda, and fits lambda to LLM responses in signaling-bandit mushroom and everyday (housing, dining) settings.

Key results

• RLHF improved both truthfulness and helpfulness for Mixtral (pre vs post RLHF) and LLaMA 2 70B, and in Experiment 2 made Mixtral and LLaMA more balanced between values.
• Chain-of-thought raised fitted helpfulness weight lambda in 4 of 7 models in Experiment 1 with Bayes factors exceeding 1e15 for several (e.g., LLaMA 2 70B lambda 0.56 to 0.80 vanilla vs CoT).
• GPT-4 Turbo with CoT under neutral prompts refused false-but-helpful utterances yet became steerable by helpfulness/honesty prompts in a human-like manner, while smaller models were not steerable.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought reasoning causes models to choose more false-but-helpful utterances (more lies told, and lies more often helpful), despite reasoning being expected to improve honesty.
• Pre-RLHF Mixtral prefers helpfulness while post-RLHF Mixtral prefers honesty, yet post-RLHF Mixtral becomes less truthful when prompted for honesty.

Limitations

• Controlled synthetic domains may not generalize to real-world contexts.
• Binary endorsement paradigm replaces human slider, and some Llama 2 runs were near-uniform and excluded from analysis.
• Human comparisons are complicated by differing social and physical groundings.

Experimental setup

Datasets: signaling bandits mushroom stimuli housing and dining cover-story stimuli

Benchmarks: signaling bandits mushroom task housing and dining everyday settings

Metrics: fraction truthful fraction helpful fitted lambda (utility weight) speaker optimality beta_S Bayes factor endorsement curves

Baselines: GPT-3.5 Turbo GPT-4 LLaMA 2 70B LLaMA 2 70B chat Mixtral 8x7B Mixtral 8x7B Instruct human participants

Builds on: Rational Speech Acts signaling bandits chain-of-thought prompting Gricean maxims

Closest papers by meaning

• HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.89)
• Alignment for Honesty (NeurIPS 2024 · similarity 0.89)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.89)
• Eliciting Human Preferences with Language Models (ICLR 2025 · similarity 0.88)
• More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.88)
• Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.87)