Research map / Large language models / Agents, code and math
A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
Qingchuan Ma, Yuexiao Ma, Yongkang Xie, Tianyu Xie, Xiawu Zheng, Rongrong Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remap symbol tokens in symbolic tasks (ablation) | GPT-5 · A2RBench symbolic tasks | accuracy: 41.3 -> 23.6 | Hurt |
| remap symbol tokens in symbolic tasks (ablation) | Claude-Sonnet-4.5 · A2RBench symbolic tasks | accuracy: 34.2 -> 32.5 | No clear effect |
| augment seed tasks with expanded input variations (V1-9) | solver LLMs · A2RBench | accuracy: Gemini3-Pro 39.8 -> 41.0 | No clear effect |
| use 3D tasks instead of 2D tasks | solver LLMs · A2RBench | accuracy | Improved |
| use highly structured, higher-complexity inputs (variation V4) | solver LLMs · A2RBench | accuracy: 54.8%, nearly triple the baseline | Improved |
| reuse validated seed code to expand tasks instead of generating new seeds | A2RBench pipeline · generation cost | cost per task (USD): $0.19 -> $0.005 | Improved |
Closest papers by meaning
- Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective (ICML 2025 · similarity 0.91)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.89)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.89)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.89)
- Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems (ICLR 2026 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
