Research map / Large language models / Alignment and preferences

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models

ICML 2025 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes metrics for logical preference consistency (transitivity, commutativity, negation invariance) for LLM pairwise judgments, evaluates ten LLMs on three tasks, and introduces REPAIR, a rank-estimation-based data refinement and augmentation method for instruction tuning.

Problem

LLM pairwise judgments over sets of items are often inconsistent (intransitive, order-dependent, or failing to invert under negation), which undermines their reliability as logical operators in decision-making and ranking algorithms.

Approach

Define set-level metrics for transitivity (acyclicity of sampled sub-graphs), commutativity (invariance to item order) and negation invariance; then refine noisy pairwise labels by estimating a global ranking (win-loss rate) and extrapolating a self-consistent comparison set, optionally augmented with negated-relation comparisons, for instruction tuning.

Key results

• Repaired Llama-3-8B data raises transitivity s_tran(5) from 81.1 (zero-shot) to 95.4 and commutativity from 70.2 to 91.2 on Summarize from Feedback, with human accuracy roughly unchanged (64.3 to 70.1).
• Adding negated-relation comparisons raises negation invariance s_neg from 60.8 to 87.9 on Summarize from Feedback.
• On PairS over SummEval, higher-transitivity models get better Spearman correlation; e.g., Phi-3-medium (s_tran 96.2, s_comm 71.0) reaches 38.9 vs. 33.5 for GPT-3.5-turbo.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought prompting generally did not improve logical consistency and sometimes lowered transitivity.
• Negation invariance was poor for many models, and adding negated-relation training data slightly hurt other consistency metrics.

Limitations

• Only a binary relation setting and a limited set of tasks and models are evaluated.
• REPAIR experiments use a single 8B model and a 200-instance test subset.
• Code and refined dataset are only referenced via an anonymous placeholder.

Experimental setup

Datasets: SummEval NovelEval CaTeRS Summarize from Feedback MS MARCO

Benchmarks: SummEval NovelEval CaTeRS Summarize from Feedback MS MARCO

Metrics: human agreement accuracy s_tran(5) transitivity s_comm commutativity s_neg negation invariance Spearman correlation (PairS) self-agreement

Baselines: Zero-shot inference Perturbed data PairS Elo rating Bradley-Terry model

Builds on: PairS Chain-of-thought prompting LoRA

Closest papers by meaning

• Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.89)
• Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language Models (NeurIPS 2024 · similarity 0.89)
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time (ICML 2025 · similarity 0.88)
• Strong Preferences Affect the Robustness of Preference Models and Value Alignment (ICLR 2025 · similarity 0.88)
• Understanding the Logic of Direct Preference Alignment through Logic (ICML 2025 · similarity 0.88)
• Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only (ICLR 2025 · similarity 0.88)