AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought

Huaxiaoyue Wang, Gonzalo Gonzalez-Pumariega, Yash Sharma, Sanjiban Choudhury

NeurIPS 2023 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add demonstrations to language-only code generation (Demo2Code vs Lang2Code)Lang2Code · Tabletop Manipulation overallunit test pass rate: 0.19 -> 0.88Improved
use Demo2Code instead of Lang2CodeLang2Code · Robotouille overallunit test pass rate: 0.29 -> 0.59Improved
use Demo2Code instead of Lang2CodeLang2Code · Robotouille overallexecution success: 0.64 -> 0.79Improved
use Demo2Code instead of Lang2CodeLang2Code · Tabletop Manipulation overallexecution success: 1.00 -> 0.99Hurt
add language instruction to demonstration-only prompting (Demo2Code vs DemoNoLang2Code)DemoNoLang2Code · Robotouille overallexecution success: 0.49 -> 0.79Improved
add language instruction to demonstration-only promptingDemoNoLang2Code · Tabletop Manipulation overallunit test pass rate: 0.81 -> 0.88Improved
use Demo2Code instead of DemoNoLang2CodeDemoNoLang2Code · Robotouille overallCode BLEU: 0.71 -> 0.74No clear effect
use Demo2Code instead of Lang2CodeLang2Code · EPIC-Kitchensusers with satisfied preference (of 7): 2 -> 5Improved
lengthen chain-of-thought from none to Full (extended CoT) (ablation)Demo2Code · Robotouille short/medium/long horizonunit test pass rateImproved
use Full chain instead of 2-steps on long-horizon tasks (ablation)Demo2Code · Robotouille long-horizonunit test pass rateImproved
use 1-step chain instead of Full on short-horizon tasks (ablation)Demo2Code · Robotouille short-horizonunit test pass rateNo clear effect
remove 10% of predicates or states from demonstrations (ablation)Demo2Code · Robotouille overallunit test pass rateNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy