AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Deductive Verification of Chain-of-Thought Reasoning

Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, Hao Su

NeurIPS 2023 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
verify each step with Natural Program decomposition instead of whole chain at onceChatGPT verifier · Overall (6 datasets)verification accuracy: 52% -> 69%Improved
step-wise verification instead of whole-chainChatGPT verifier · Dateverification accuracy: 63% -> 73%Improved
step-wise verification instead of whole-chainChatGPT verifier · Last Lettersverification accuracy: 51% -> 51%No clear effect
use minimal necessary premises instead of full premises (ablation)NP verification · Average (6 datasets)verification accuracy: 57% -> 69%Improved
one-shot prompt instead of zero-shot (ablation)NP verification with minimal premises · Average (6 datasets)verification accuracy: 53% -> 69%Improved
increase Unanimity-Plurality Voting k' from 1 to 10 (ablation)Deductive verification · AddSubaccuracy on correct-answer chains: 86% -> 92%Improved
increase k' from 1 to 10 (ablation)Deductive verification · AddSubaccuracy on wrong-answer chains: 38% -> 40%No clear effect
add deductive verification and UPV filtering to Natural Program (ablation)Natural Program · GSM8Kfinal answer accuracy: 87.05% -> 86.01%Hurt
add deductive verification and UPV filtering to Natural Program (ablation)Natural Program · AddSubfinal answer accuracy: 93.67% -> 93.54%No clear effect
Natural Program prompting instead of CoT + VotingChatGPT · Last Lettersfinal answer accuracy: 81.60% -> 92.98%Improved
Natural Program prompting instead of CoT + VotingChatGPT · GSM8Kfinal answer accuracy: 87.62% -> 87.05%No clear effect
fine-tune Vicuna-13B on deductive verification datasetVicuna-13B · Overall (6 datasets)verification accuracy: 57% -> 66%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy