Research map / Large language models / Agents, code and math
Argumentative Large Language Models for Explainable and Contestable Claim Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArgLLMs, which use LLMs to build quantitative bipolar argumentation frameworks (QBAFs) for claim verification, evaluated with DF-QuAD gradual semantics, yielding explainable, contestable decisions with accuracy comparable to CoT and direct prompting.
Problem
LLM decisions are hard to explain faithfully and cannot be reliably contested or corrected by external agents, which limits their use in decision-making such as claim verification.
Approach
Augment LLMs with formal argumentation: the LLM generates supporting and attacking arguments for a claim, assigns each an intrinsic strength zero-shot, and a gradual semantics (DF-QuAD) computes the claim's strength deterministically, giving a faithful QBAF explanation that can be contested by editing arguments or base scores, with formal contestability properties.
Key results
- On TruthfulClaim, the Est. Base Arg (D=1) ArgLLM with Mixtral reached 0.81 accuracy versus 0.77 for direct question and 0.76 for chain-of-thought.
- ArgLLMs perform comparably to chain-of-thought across models and datasets, with no accuracy loss from adding explainability.
- On MedClaim with GPT-4o, direct question (0.85) and chain-of-thought (0.85) outperformed the best ArgLLM variant (0.83).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ArgLLM with estimated claim base score, depth 1, rather than the strongest reported baseline | Mixtral · TruthfulClaim | accuracy: 0.79 -> 0.81 | Helped |
| Use ArgLLM with estimated claim base score, depth 2, rather than the strongest reported baseline | Mixtral · StrategyClaim | accuracy: 0.77 -> 0.70 | Hurt |
| Use ArgLLM with estimated claim base score, depth 1, rather than the strongest reported baseline | GPT-4o · MedClaim | accuracy: 0.85 -> 0.83 | Hurt |
| Use estimated rather than neutral (0.5) base score for the claim (ablation) | ArgLLM, TruthfulClaim · TruthfulClaim | accuracy: 0.74 -> 0.81 | Helped |
| Increase argumentation depth from 1 to 2 (ablation) | ArgLLM with neutral claim base score, TruthfulClaim · TruthfulClaim | accuracy: 0.65 -> 0.67; 0.72 -> 0.72; 0.74 -> 0.78 | Mixed |
| Increase argumentation depth from 1 to 2 (ablation) | ArgLLM with estimated claim base score, TruthfulClaim · TruthfulClaim | accuracy: 0.76 -> 0.75; 0.81 -> 0.78 | Mixed |
| Use estimated rather than neutral (0.5) base score for the claim (ablation) | ArgLLM, StrategyClaim · StrategyClaim | accuracy: 0.58 -> 0.62; 0.66 -> 0.68; 0.66 -> 0.74 | Mixed |
| Increase argumentation depth from 1 to 2 (ablation) | ArgLLM with neutral claim base score, StrategyClaim · StrategyClaim | accuracy: 0.58 -> 0.61; 0.62 -> 0.66; 0.56 -> 0.59 | Mixed |
| Use estimated rather than neutral (0.5) base score for the claim (ablation) | ArgLLM, MedClaim · MedClaim | accuracy: 0.62 -> 0.71 | Helped |
| Increase argumentation depth from 1 to 2 (ablation) | ArgLLM with neutral claim base score, MedClaim · MedClaim | accuracy: 0.50 -> 0.51; 0.59 -> 0.56; 0.62 -> 0.65 | Mixed |
| Replace DF-QuAD with QEM semantics (ablation) | ArgLLM · TruthfulClaim | accuracy | Hurt |
| Use QEM instead of DF-QuAD (ablation) | ArgLLM · StrategyClaim | accuracy: 0.69 -> 0.70; 0.70 -> 0.71; 0.75 -> 0.74 | Mixed |
Surprising findings
- Even with GPT-4o, ArgLLM variants underperform direct questioning and chain-of-thought on MedClaim, suggesting limited zero-shot argument generation and estimation in powerful models.
Limitations
- Accuracy comparable to baselines rather than better; ArgLLM variants trail on some model-dataset pairs.
- Claims were created from QA pairs using LLMs with manual checking, and only subsets (500 per dataset) were used due to cost.
- Intrinsic strengths are elicited zero-shot by the LLM with no validation against human argument-quality judgments; human evaluation is planned future work.
Experimental setup
Datasets: TruthfulQA StrategyQA MedQA
Benchmarks: TruthfulClaim StrategyClaim MedClaim
Metrics: accuracy
Baselines: Direct Question Estimated Confidence Chain-of-Thought
Builds on: chain-of-thought prompting quantitative bipolar argumentation frameworks DF-QuAD quadratic energy model
Closest papers by meaning
- Neurosymbolic Learning for Inference-Time Argumentation (NeurIPS 2026 · similarity 0.87)
- Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.87)
- Complexity of Credulous and Skeptical Acceptance in Epistemic Argumentation Framework (AAAI 2024 · similarity 0.87)
- Redefining ABA+ Semantics via Abstract Set-to-Set Attacks (AAAI 2024 · similarity 0.87)
- Reasoning Structure of Large Language Models (ICML 2026 · similarity 0.86)
- DeLLMa: Decision Making Under Uncertainty with Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.