Research map / Large language models / Agents, code and math
CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
Daniel Kaiser, Arnoldo Frigessi, Ali Ramezani-Kebrya, Benjamin Ricaud
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase puzzle difficulty d from 1 to 10 (ablation) | GPT-5 · CogniLoad | accuracy (%): 100 -> 82 | Hurt |
| increase puzzle difficulty d from 1 to 10 (ablation) | Qwen3-1.7B · CogniLoad | accuracy (%): 33 -> 11 | Hurt |
| increase statement length N from 20 to 250 (ablation) | Gemini-2.5-Pro · CogniLoad | accuracy (%): 99 -> 21 | Hurt |
| increase statement length N from 20 to 250 (ablation) | GPT-5 · CogniLoad | accuracy (%): 100 -> 76 | Hurt |
| increase statement length N from 20 to 250 (ablation) | Gemini-2.5-Flash · CogniLoad | accuracy (%): 99 -> 14 | Hurt |
| raise distractor ratio to mid range (rho 25-50%) (ablation) | most models · CogniLoad | accuracy | Hurt |
| add quadratic rho term to logistic GLM (ablation) | accuracy GLM · CogniLoad, 22 models | AIC: 1032.432 -> 1019.177 (DeepSeek-R1-0528) | Improved |
| add quadratic rho term to logistic GLM (ablation) | accuracy GLM · DeepSeek-R1-Distill-Qwen-1.5B | AIC: 11582.864 -> 11584.667 | No clear effect |
| add quadratic rho term to logistic GLM (ablation) | accuracy GLM · o3-2025-04-16 | AIC: 692.757 -> 692.356 | No clear effect |
Closest papers by meaning
- Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning (NeurIPS 2025 · similarity 0.90)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
- When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.88)
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models (ICML 2026 · similarity 0.88)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
