Research map / Large language models / Agents, code and math
Do Large Language Models Know What They Are Capable Of?
Casey O. Barkan, Sidney Black, Oliver Sourbut
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale up to newer or larger LLMs within a family | Llama, GPT and Claude models · BigCodeBench in-advance confidence | AUROC | Mixed |
| add in-context experience of past successes and failures | Claude Sonnet models and GPT 4.5 · resource acquisition contracts | profit | Improved |
| add in-context experience of past successes and failures | GPT 4.1 · resource acquisition contracts | profit | No clear effect |
| use reasoning LLMs instead of non-reasoning LLMs | SWE-Bench Verified confidence estimation · SWE-Bench Verified | overconfidence and AUROC | No clear effect |
| progress through more steps of a multi-step task | Claude Sonnet models · SWE-Bench Verified | overconfidence | Hurt |
| progress through more steps of a multi-step task | OpenAI models except GPT 5.1 · SWE-Bench Verified | AUROC | Improved |
Closest papers by meaning
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.92)
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.91)
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.90)
- Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
