Research map / Large language models / Alignment and preferences

Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows LLMs can rank wrong answers by wrongness (score-based LLM-as-judge best, up to 70.9% accuracy) and that DPO on such wrong-over-wrong preference pairs makes LLaMA3-8B less wrong, sometimes more correct, and better calibrated.

Problem

Alignment usually needs correct answers, which are unavailable for challenging tasks; the paper asks whether LLMs can order incorrect answers by wrongness and whether aligning on such wrong-over-wrong preferences helps.

Approach

Elicit preferences between pairs of wrong answers using self-consistency, token NLL, pairwise LLM comparison (with order-flip consistency), and score-based LLM-as-judge with score-margin filtering, then align with DPO using only wrong answers as chosen/rejected pairs.

Key results

• Score-based elicitation with GPT-4o and margin M10 reaches Acc_WoW 0.709 overall, versus 0.474 for the heuristic baseline.
• DPO on wrong-over-wrong pairs improves LLaMA3-8B's average wrongness proxy by 0.074, accuracy by 0.045 and calibration (-ΔECE) by 0.044 across datasets.
• Pairwise comparison with consistency checks gives the best calibration gain (-ΔECE = 0.094 on average).

Measured effects reported in the paper

Surprising findings

• Wrong-over-wrong alignment raises accuracy on correct answers despite training only on incorrect answers.
• Self-evaluation is weaker: GPT-4o's accuracy drops about 21% on its own answers versus other generators' answers, while weak LLaMA3-8B judging GPT-4o answers reaches 0.649.

Limitations

• Wrongness proxies are imperfect and depend on ground truth, so evaluation is silver rather than true.
• Results are sensitive to hyperparameters, and alternative preference optimizers (SimPO, IPO) collapse without tuning.
• No benefit observed on domains without clear right-wrong distinctions (MedMCQA, SciBench).

Experimental setup

Datasets: Knowledge Crosswords NLGraph shortest path FActScore Bio Generation COM2 Hellaswag NLGraph maximum flow Chess Puzzle SciBench MedMCQA Code-Alpaca FLAN LIMA

Benchmarks: Knowledge Crosswords NLGraph shortest path FActScore Bio Generation COM2 Hellaswag NLGraph maximum flow SciBench MedMCQA

Metrics: Acc_WoW average wrongness proxy p_wrong accuracy expected calibration error

Baselines: Heuristic (response length) Consistency-based Logits-based Pairwise comparison Score-based Oracle preferences Right-over-wrong DPO

Builds on: DPO self-consistency LLM-as-a-judge FActScore Vera

Closest papers by meaning

• On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.90)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.89)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
• Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR 2024 · similarity 0.89)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)