Research map / Large language models / Reasoning and chain of thought
Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PODA-TPCL, which generates CoT rationales covering both correct and incorrect options plus premise-based counterfactual contexts with GPT-4, and trains LLaMA2-7B, Mistral-7B and LLaMA3-8B with an added thought-path contrastive loss for logical reading comprehension.
Problem
Logical reading comprehension requires deducing the correct answer from context, question and options, but existing CoT rationales only analyze correct options and rule-based counterfactual augmentation yields low-diversity, poorly coherent contexts that mismatch the options.
Approach
Premise-oriented augmentation: summarize premises, identify each option's relation (supported/contradicted/unrelated) to premises, and use masked-NLI style prompts to generate counterfactual premises and contexts for incorrect options; then a Bradley-Terry style contrastive loss pulls similar thought-path pairs together and pushes dissimilar pairs apart between original and counterfactual samples.
Key results
- TPReasoner with Mistral-7B reaches 71.17% test accuracy on ReClor and 60.28% on LogiQA 2.0 test, the best open-model results in the paper.
- Removing TPCL drops accuracy by roughly 1-3 points across the three base models, and removing counterfactual data drops it further.
- On generated-data quality, GPT-4o rates PODA contexts at 4.22 overall versus 2.91 for the rule-based LReasoner, with diversity 3.18 versus 1.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add thought-path contrastive learning (TPCL) (ablation) | TPReasoner with LLaMA3-8B · ReClor Test | accuracy: 69.30 -> 70.97 | Helped |
| add counterfactual data (CD) (ablation) | TPReasoner with LLaMA3-8B, without TPCL · ReClor Test | accuracy: 66.10 -> 69.30 | Helped |
| add wrong-option analyses (WOA) in CoT rationales (ablation) | TPReasoner with LLaMA3-8B, without TPCL and CD · ReClor Test | accuracy: 64.03 -> 66.10 | Helped |
| use TPReasoner instead of LLaMA3-8B-logicot | LLaMA3-8B · ReClor Test | accuracy: 62.65 -> 70.97 | Helped |
| use TPReasoner instead of LLaMA3-8B-logicot | LLaMA3-8B · LogiQA 2.0 Test | accuracy: 54.07 -> 58.78 | Helped |
| generate counterfactual contexts with PODA instead of LReasoner | counterfactual context generation · Counterfactual data quality evaluation | Coherence: 2.98 -> 4.61 | Helped |
| generate counterfactual contexts with PODA instead of LReasoner | counterfactual context generation · Counterfactual data quality evaluation | Diversity: 1.08 -> 3.18 | Helped |
| include analyses of incorrect options in CoT rationales | PODA instead of LogiCoT · CoT rationale evaluation | Completeness: 3.38 -> 4.46 | Helped |
| use TPReasoner instead of LLaMA2-7B-logicot | LLaMA2-7B · ReClor Test | accuracy: 50.50 -> 58.63 | Helped |
| use TPReasoner instead of LLaMA2-7B-logicot | LLaMA2-7B · LogiQA 2.0 Test | accuracy: 43.19 -> 49.75 | Helped |
Surprising findings
- Counterfactual-data accuracy judged by LLaMA3-70B-Instruct (82.19%) and GPT-4 (93%) suggests about a quarter of generated labels may be wrong for weaker judges, with other models at 75-79%.
Limitations
- Counterfactual data is generated with proprietary GPT-4 and GPT-3.5 and filtered by GPT-4 verification, adding API cost and dependence.
- Some samples lack counterfactuals, requiring a two-stage training strategy.
- Human and LLM-judge evaluation covers only 200 sampled items per setting, and the accuracy check of counterfactual data used a 200-sample subset due to budget constraints.
Experimental setup
Datasets: ReClor LogiQA 2.0
Benchmarks: ReClor LogiQA 2.0
Metrics: accuracy coherence clarity relevance diversity completeness faithfulness
Baselines: RoBERTa-Large DGAN LReasoner AMR-LDA FocalReasoner LogiCoT (LLaMA2-7B-logicot) GPT-3.5 GPT-4
Builds on: LogiCoT LReasoner DPO Few-shot-CoT LoRA
Closest papers by meaning
- RATT: A Thought Structure for Coherent and Correct LLM Reasoning (AAAI 2025 · similarity 0.87)
- Fact-Driven Logical Reasoning for Machine Reading Comprehension (AAAI 2024 · similarity 0.86)
- Reasoning Scaffolding: Distilling the Flow of Thought from LLMs (ICLR 2026 · similarity 0.86)
- MixReasoning: Switching Modes to Think (ICML 2026 · similarity 0.86)
- Learning to Reason via Mixture-of-Thought for Logical Reasoning (ICLR 2026 · similarity 0.86)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.