Published by Latentic Labs with scime atlas
Special edition · LLM agents & reasoning · what 3,110 papers read in full say
1.0% of accepted papers in 2023, 10.3% in 2026
2,708 of the papers accepted at ICML, NeurIPS, ICLR, CVPR and AAAI in 2026 work on reasoning, chain of thought, LLM agents, code or math, up from 118 in 2023.
scime atlas groups all 108,967 accepted papers of these five conferences (2016–2026) into topics by meaning. Three of those topics are about making large language models reason and act: reasoning and chain of thought, agents, code and math, and LLM agents (several agents, tools and environments). Together they held 0.1% of the papers in 2016 and 10.3% this year.
For 3,110 of these papers, mostly from 2024–2026, scime atlas has read the full text with AI and recorded each change a paper tested and what it measured: 15,539 findings in all. This edition asks what they say: which ideas keep helping, where they do not, what surprised the authors and what they say is still open.
Accepted papers per year in the three topics, all five conferences.
AI-assisted. Topics come from the atlas clustering; what each paper tried, found and reports as limits was extracted by AI from its full text. Nothing here was checked by hand and some lines are wrong; open the paper before you cite it. Published papers report what worked more often than what did not, so read the shares as "what gets reported".
In numbers
2023 → 2026, by topic
2026 papers by conference
Inside
Try it on your idea
Describe your own agent or reasoning idea in scime atlas to see the closest of 108,967 papers and what similar changes did in them.
Writing it up? scime draft is a free LaTeX editor with AI, and it imports OpenAI Prism projects.
What keeps helping
1,396 papers that added something and measured itWe sorted every change these papers added to a model, prompt or training run (removals and swaps are left out, because they read the other way round) into twelve families of ideas, and counted, per paper, how often the paper reports that the change helped, made no clear difference or made things worse. Across all of them, papers report that what they added helped in 68% of cases.
Supervised fine-tuning and distillation
269 papers tested it · helped in 76% of what they report
- helped Fine-tune the ego vehicle using ChatScene-generated adversarial scenarios rather than the strongest baseline scenarios on SAC-trained ego vehicle — 8 base scenarios (average), post-finetuning test · collision rate (0.135 -> 0.043; an additional 9%) — ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles CVPR 2024 · 79 citations
- no clear effect Fine-tune on successful circuit designs on GPT-3.5 — Analog circuit benchmark · Number solved (10 -> 10) — AnalogCoder: Analog Circuit Design via Training-Free Code Generation AAAI 2025 · 55 citations
Chain-of-thought and reasoning traces
188 papers tested it · helped in 62% of what they report
- helped Start from “Let's think step by step.” rather than the empty string or “Let's solve the problem.” on OPRO with PaLM 2-L scorer — Prompt optimization · accuracy — Large Language Models as Optimizers ICLR 2024 · 93 citations
- worse Add CoT-decoding to greedy decoding on PaLM-2 Large — Sports Understanding · accuracy (58.8 -> 58.0) — Chain-of-Thought Reasoning Without Prompting NeurIPS 2024 · 36 citations
Sampling, voting and test-time compute
119 papers tested it · helped in 72% of what they report
- helped increase sampling from Pass@1 to Pass@5 on GPT-3.5 with one-shot-relevant — RobustAPI · API Misuse Rate (39.06% -> 21.98%) — Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation AAAI 2024 · 63 citations
- worse Further reduce SoM screenshot resolution to down-sampling ratio 0.2 on GPT-4V (Set-of-Marks) — OSWorld subset (10%) · success rate — OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments NeurIPS 2024 · 39 citations
Reinforcement learning (GRPO, PPO, RLVR, DPO)
118 papers tested it · helped in 80% of what they report
- helped Add PRM to SFT during PPO training on WizardMath-SFT — GSM8K and MATH · pass@1 (77.4 -> 81.7; 35.6 -> 39.9) — WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct ICLR 2025 · 28 citations
- worse Increase single-PPO model size from GPT2-Large to GPT2-XL on single-agent PPO — IMDB Review · KL divergence — Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning NeurIPS 2024 · 8 citations
Tools, retrieval and code execution
101 papers tested it · helped in 77% of what they report
- helped Add retriever-aware training with ground-truth API documents on Gorilla — Torch Hub · accuracy (59.13 -> 67.20) — Gorilla: Large Language Model Connected with Massive APIs NeurIPS 2024 · 93 citations
- worse Add human feedback and retrieved code examples on Llama 2 — LaMPilot-Bench · Driving Score (39.6 -> 21.4 (-18.2)) — LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs CVPR 2024 · 48 citations
Search and planning (tree search, MCTS)
73 papers tested it · helped in 76% of what they report
- helped add AST-WT to Planner1 on Planner1 — VerilogEval-Human v2 · pass-rate (66.7% -> 78.2%; 11.5%) — VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool AAAI 2025 · 46 citations
- no clear effect Add LLaDA re-planning to GPT-Driver on GPT-Driver — nuScenes Boston validation, 1s · L2 (m) (0.27 -> 0.27) — Driving Everywhere with Large Language Model Policy Adaptation CVPR 2024 · 35 citations
Reflection and self-correction
72 papers tested it · helped in 81% of what they report
- helped Add short-term reflection on ReEvo ACO heuristic search — TSP50 ACO random walk · objective (12.08 ± 7.15 -> 6.53 ± 0.60) — ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution NeurIPS 2024 · 53 citations
- worse Add reflections to insight construction on ExpeL — HotpotQA · success rate (%) (39.0 -> 29.0) — ExpeL: LLM Agents Are Experiential Learners AAAI 2024 · 123 citations
Curricula and data selection
47 papers tested it · helped in 53% of what they report
- helped Adapt the training compression level to question difficulty on C3oT, GSM8K — GSM8K · accuracy (36.92 -> 40.85) — C3oT: Generating Shorter Chain-of-Thought Without Compromising Effectiveness AAAI 2025 · 10 citations
- worse Increase IaC problem difficulty on Evaluated LLMs — IaC-Eval · average score — IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs NeurIPS 2024 · 14 citations
Length and token budgets
46 papers tested it · helped in 55% of what they report
- helped Increase verification share as total query budget grows on InternLlama generator-verifier pipeline — Blend Shape · photometric loss — BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing CVPR 2025 · 6 citations
- no clear effect Share/compress the KV cache to a budget of s=4 on Huginn-0125 — MT-Bench (1-turn) · score (5.693 -> 5.856) — Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach NeurIPS 2025 · 5 citations
Reward models and verifiers
44 papers tested it · helped in 75% of what they report
- helped Add PRM to SFT during PPO training on WizardMath-SFT — GSM8K and MATH · pass@1 (77.4 -> 81.7; 35.6 -> 39.9) — WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct ICLR 2025 · 28 citations
- no clear effect Add RM-weighted voting to combine PRM scores and majority voting on PRM and majority-vote search — MATH test subset · accuracy — Let's Verify Step by Step ICLR 2024 · 31 citations
Memory and long context
43 papers tested it · helped in 71% of what they report
- helped Add the memory unit on Mobile-Agent-v2 — Ablation evaluation, basic instructions · SR (77.3 -> 88.6) — Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration NeurIPS 2024 · 26 citations
- no clear effect Add passive memory vaccines on LLM agents — Harmless unusual instructions · agent cooperation (87.1% to 87.5%) — Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems AAAI 2025 · 8 citations
Several agents, debate and roles
15 papers tested it · helped in 84% of what they report
- helped Search agent designs in code with Meta Agent Search rather than use LLM Debate on GPT-3.5 agents — MGSM · accuracy (39.0 ± 3.4 -> 53.4 ± 3.5) — Automated Design of Agentic Systems ICLR 2025 · 8 citations
- no clear effect Increase the number of debate rounds on Qwen2.5-7B-Instruct decentralized MAD — CNN/DailyMail · ROUGE-L — Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? NeurIPS 2025 · 4 citations
Each paper counts once per family, split evenly over its findings of that kind. Bars: green helped, olive mixed, grey no clear effect, red worse.
Free for signed-in readers
Read the rest of this edition
Four more pages: 24 negative and null results from the most cited papers, what surprised the authors, the benchmarks and baselines everyone uses, and what the papers say is still open. Sign in with a free scime account to keep reading.
- C When it did not help
- D What surprised the authors
- E The benchmarks everyone uses
- F What is still open
Already signed in? Reload this page. Accounts are free; we only use your email to sign you in and, if you agree, to tell you about new editions.
About the data
how this edition was madePapers. All 108,967 accepted papers of ICML, NeurIPS, ICLR, CVPR and AAAI from 2016 to 2026 in scime atlas, grouped into topics by meaning (an open-source embedding model and clustering; topic names written with AI). This edition covers three topics: Reasoning and chain of thought, Agents, code and math, LLM agents.
Full texts. For 3,110 of these papers an AI model read the full text and wrote a structured card (problem, idea, results, surprises, limitations, benchmarks, baselines) and a list of findings: each change the paper tested, on what, the metric, the reported number and whether it helped. These were not checked by hand. Each paper links to its page on scime atlas, which links to the paper itself.
Counting. Shares of findings count each paper once (split evenly over its findings of a kind). Only changes that add or use something are counted in "what keeps helping"; removals and swaps are left out because their effect reads the other way round. Citation counts are from OpenAlex (cited_by_count) and favor older papers.
Caveat. Published papers report successes more often than failures, so a high "helped" share means "often reported as helping", not "likely to help you". Corrections: contact@latenticlabs.com. Not affiliated with or endorsed by any of the conferences.
← All editions · © 2026 Latentic Labs