Research map / Large language models / Agents, code and math
LogiConBench: Benchmarking Logical Consistencies of LLMs
Zheng CHEN, Chuan Zhou, Fengxiang Cheng, Yip Tin Po, Fenrong Liu, Yisen Wang, Jiajun Chai, Xiaohan Wang, Guojun Yin, Wei Lin, Bo Li, Haoxuan Li and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3-shot examples with reasoning paths instead of zero-shot (ablation) | 14 LLMs · Task 1 discriminative | average accuracy: 59.21% -> 67.58% | Improved |
| add 3-shot examples instead of zero-shot (ablation) | 14 LLMs · Task 1 discriminative | average accuracy: 59.21% -> 65.70% | Improved |
| increase statements from k=2 to k=5 | 14 LLMs · Task 1 discriminative | average accuracy: 66.87% -> 59.59% | Hurt |
| add 3-shot examples with reasoning paths instead of zero-shot (ablation) | o3-mini · Task 1, 2 statements | overall accuracy: 49.80% -> 51.70% | Improved |
| add 3-shot examples with reasoning paths instead of zero-shot (ablation) | gpt-5 · Task 2, 5 statements | exact accuracy: 0.073 -> 0.155 | Improved |
| add 3-shot examples with reasoning paths instead of zero-shot (ablation) | gpt-5 · Task 2, 3 statements | exact accuracy: 0.439 -> 0.505 | Improved |
| restrict to short-reasoning-path samples | gpt-5 · Task 2 easy samples | exact accuracy: 34% -> 58% | Improved |
| restrict to short-reasoning-path samples | grok-4-fast · Task 2 easy samples | exact accuracy: 12% -> 31% | Improved |
| restrict to short-statement-length samples | gpt-5 · Task 2 easy samples | exact accuracy: 34% | No clear effect |
| evaluate on hard samples differing by one label | frontier LLMs · Task 1 hard samples | accuracy: around 80% | Hurt |
| use counterfactual or commonsense propositions | LLMs · Task 2 | performance | No clear effect |
| rewrite statements into human-style paraphrases | small vs large LLMs · Task 2 | performance | Mixed |
Closest papers by meaning
- Logical Consistency of Large Language Models in Fact-Checking (ICLR 2025 · similarity 0.91)
- LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision (ICLR 2026 · similarity 0.89)
- Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus (NeurIPS 2024 · similarity 0.89)
- ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.88)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
