Research map / Large language models / Alignment and preferences
Aligning Large Language Models for Faithful Integrity Against Opposing Argument
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFICE, which uses Bilateral Confidence Estimation (regressor on intermediate-layer states plus cumulative probability ratios) to build a conversational preference dataset from MMLU and aligns LLMs with DPO so they keep faithful answers against opposing arguments and correct wrong ones.
Problem
LLMs are easily swayed by opposing user arguments, abandoning correct statements, and fail to correct incorrect statements when faced with faithful arguments; the paper targets maintaining faithful integrity in multi-turn conversations.
Approach
Estimate question- and answer-level confidence with a white-box method (semantic-entropy regressor over intermediate hidden states, adjusted by length-normalized cumulative probability ratios over beam samples), then use that confidence to select which of five stance-graded responses are preferred vs. dispreferred for DPO training.
Key results
- AFICE raises Vicuna average accuracy to 0.606 (math/logic/commonsense/generic averages 0.606 commonsense, 0.672 generic) versus 0.515 base and 0.580 for semantic entropy on the commonsense table.
- On LLaMA-3 8B, AFICE reaches 0.709 commonsense average and 0.742 generic average, against 0.509 and 0.598 for the base model.
- The authors report BCE has higher AUROC and lower ECE than the four baselines on MMLU, though the exact values are only shown in a figure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AFICE rather than Predictive Entropy | Vicuna 7B · GSM8K | accuracy: 0.616 -> 0.623 | Helped |
| Apply AFICE rather than Predictive Entropy | Vicuna 7B · PrOntoQA | accuracy: 0.723 -> 0.744 | Helped |
| Apply AFICE rather than Predictive Entropy | Vicuna 7B · Commonsense average | accuracy: 0.573 -> 0.606 | Helped |
| Apply AFICE rather than Predictive Entropy | Vicuna 7B · Generic average | accuracy: 0.622 -> 0.672 | Helped |
| Apply AFICE rather than Semantic Entropy | LLaMA-3 8B · PrOntoQA | accuracy: 0.733 -> 0.752 | Helped |
| Apply AFICE rather than Semantic Entropy | LLaMA-3 8B · Commonsense average | accuracy: 0.640 -> 0.709 | Helped |
| Apply AFICE rather than Semantic Entropy | LLaMA-3 8B · Generic average | accuracy: 0.695 -> 0.742 | Helped |
| Add P(True) to AFICE (ablation) | Vicuna 7B · Commonsense average | accuracy: 0.546 -> 0.606 | Helped |
| Add P(True) to AFICE (ablation) | LLaMA-3 8B · PrOntoQA | accuracy: 0.518 -> 0.752 | Helped |
| Add Predictive Entropy to AFICE (ablation) | Vicuna 7B · Commonsense average | accuracy: 0.573 -> 0.606 | Helped |
| Add Predictive Entropy to AFICE (ablation) | LLaMA-3 8B · Commonsense average | accuracy: 0.660 -> 0.709 | Helped |
| Use cumulative probability ratios (set γ=0.3 rather than γ=0) (ablation) | AFICE confidence estimation · MMLU | AUROC | Helped |
Surprising findings
- Verbalized confidence from the base models is strongly overconfident, with the model answering correctly in nearly all cases when told it is correct and failing when told it is wrong on some benchmarks.
- Setting the confidence-adjustment hyperparameter gamma to 0 (no cumulative probability ratio) gives lower AUROC than moderate values such as 0.3.
Limitations
- Experiments only on 7–8B open models; larger models were not tested.
- The regressor must be retrained because fine-tuning changes internal states.
- Multinomial beam sampling produces broader distributions than beam search, which may reduce representativeness of the cumulative probability ratios.
Experimental setup
Datasets: MMLU GSM8K PrOntoQA StrategyQA CommonsenseQA 2.0 Creak BIG-Bench Hard tasks
Benchmarks: GSM8K PrOntoQA StrategyQA CommonsenseQA 2.0 Creak BIG-Bench Hard tasks
Metrics: accuracy average accuracy across LLM Correct and LLM False conversati AUROC ECE
Baselines: Verbalization Semantic Entropy P(True) Predictive Entropy
Builds on: Direct Preference Optimization Semantic Entropy P(True) Predictive Entropy
Closest papers by meaning
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.91)
- Alignment for Honesty (NeurIPS 2024 · similarity 0.91)
- HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.90)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.90)
- Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only (ICLR 2025 · similarity 0.89)
- Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.