Research map / Large language models / Retrieval and knowledge
Logical Consistency of Large Language Models in Fact-Checking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces logical fact-checking datasets over knowledge graphs and consistency measures for negation, conjunction, and disjunction in retrieval-augmented LLMs, then improves consistency via QLoRA supervised fine-tuning on primitive operators.
Problem
LLMs give inconsistent answers to logically equivalent or opposite propositional queries (negation, conjunction, disjunction), a gap not addressed by paraphrase-based consistency work, especially for complex queries in fact-checking.
Approach
Defines logical consistency for LLM binary responses under logical operators and DNF decomposition, builds KG-grounded fact-checking benchmarks (FreebaseLFC, NELLLFC, WikiLFC), and shows that QLoRA fine-tuning on simple/primitive-operator facts from FreebaseLFC improves consistency on complex and out-of-distribution facts.
Key results
- Fine-tuning Llama-2-13B on FreebaseLFC raises average accuracy from 0.69 to 0.88 and consistency from 0.74 to 0.88 across datasets.
- Fine-tuning improves consistency by about 14% on average across datasets including untrained NELLLFC and WikiLFC.
- Adding KG context raises Llama-2-7B simple-fact consistency from 0.03 to 0.77 on FreebaseLFC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| supervised fine-tuning on primitive operator facts (QLoRA) | Llama2-13B, Llama2-7B, Gemma-2B · FreebaseLFC, NELLLFC, WikiLFC (simple and one-operator facts) | logical consistency: average consistency 0.74 -> 0.88 | Helped |
| supervised fine-tuning on FreebaseLFC only | Llama2-13B, Llama2-7B, Gemma-2B · NELLLFC and WikiLFC (out-of-training datasets) | accuracy and consistency: e.g., Llama2-13B NELL p,¬p consistency 0.76 -> 0.93 | Helped |
| supervised fine-tuning | disjunction fact p∨q · FreebaseLFC, Llama2-7B and Gemma-2B | logical consistency: Llama2-7B 0.83 -> 0.77; Gemma-2B 0.91 -> 0.90 | Hurt |
| supervised fine-tuning | p∧(q∨r) complex fact · NELLLFC, Llama2-13B | accuracy: 0.42 -> 0.38 | Hurt |
| adding KG context to the query (ablation) | Llama2-7B base model · FreebaseLFC simple facts | accuracy and logical consistency: accuracy 0.51 -> 0.88; consistency 0.03 -> 0.77 | Helped |
| zero-shot instruction prompting (ablation) | Llama2-13B base model · FreebaseLFC simple facts | logical consistency: 0.81 -> 0.79 | Hurt |
| supervised fine-tuning vs zero-shot instruction prompting (ablation) | Llama2-13B on simple facts · FreebaseLFC, NELLLFC, WikiLFC | logical consistency: FreebaseLFC prompt 0.79 vs FT 0.86 | Helped |
| reduced context length (200 tokens vs 1000) (ablation) | Llama2/Gemma base models · FreebaseLFC | consistency | No clear effect |
| reduced context length (200 tokens) combined with fine-tuning (ablation) | fine-tuned models · FreebaseLFC | accuracy and consistency | Helped |
| parameter-efficient fine-tuning (QLoRA) vs full fine-tuning (ablation) | Llama2-7B · training time per epoch | time per epoch: 4.43 hours vs 12.52 hours | Helped |
Surprising findings
- Zero-shot instruction prompting does not improve, and often reduces, logical consistency relative to no instruction on base models.
- Chain-of-thought prompting gave no improvement over fine-tuning, per the authors.
Limitations
- Only binary (yes/no) responses are considered.
- Uses KG contexts and assumes complete KGs; no link prediction or multi-hop reasoning evaluated.
- Does not measure whether consistency improvements transfer to general reasoning benchmarks.
Experimental setup
Datasets: FB15K NELL WikiKG90Mv2 FreebaseLFC
Benchmarks: FreebaseLFC NELLLFC WikiLFC
Metrics: accuracy logical consistency
Baselines: Llama-2-7B zero-shot prompting Llama-2-13B zero-shot prompting Gemma-2B zero-shot prompting
Builds on: LoRA QLoRA Query2Box
Closest papers by meaning
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.89)
- CITE: Anytime Valid Statistical Inference in LLM Self-Consistency (NeurIPS 2026 · similarity 0.89)
- Content-free Logical Modification of Large Language Model by Disentangling and Modifying Logic Representation (AAAI 2025 · similarity 0.89)
- Calibrating Reasoning in Language Models with Internal Consistency (NeurIPS 2024 · similarity 0.88)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.88)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.