Research map / Large language models / Agents, code and math
OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OMEGA, a programmatically generated math benchmark testing exploratory, compositional, and transformative out-of-distribution generalization, and finds frontier LLMs degrade with complexity while RL helps mainly in-distribution and exploratory settings.
Problem
Frontier LLMs with long chain-of-thought reach high accuracy on Olympiad-level math benchmarks but may rely on narrow strategies and fail on problems needing novel reasoning; existing math datasets are too coarse or too easy to isolate generalization types.
Approach
Template-based problem generators (40+ families across six domains) with matched train/test splits along three Boden-inspired axes (exploratory, compositional, transformative), with answers verified by symbolic, numerical, or graphical methods.
Key results
- Frontier models (DeepSeek-R1, Claude-3.7-Sonnet, o3-mini, o4-mini) show accuracy near zero at the highest complexity levels on OMEGA, with performance declining monotonically with complexity.
- RL with GRPO on Qwen2.5-7B-Instruct gives large in-distribution gains (e.g., +61 points on ID Zebra Logic) and smaller OOD gains (e.g., +53 OOD on Zebra Logic, +8 pp OOD on geometry).
- After RL on individual skills, compositional OOD accuracy improves only marginally (e.g., about +6%), and transformative OOD accuracy stays near 0% in most settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GRPO to low-complexity problems | Qwen2.5-7B-Instruct · Exploratory generalization, Zebra Logic ID and OOD | accuracy: +61 points on ID and 53 points on OOD | Helped |
| Apply GRPO to low-complexity problems | Qwen2.5-7B-Instruct · Exploratory generalization, Geometry ID and OOD | accuracy: +31 pp on ID and just +8 pp on OOD | Mixed |
| Train with GRPO on levels 1–2 | Qwen2.5-7B-Instruct · Arithmetic GCD, level 3 | accuracy: 6% to 10% | Helped |
| Expand GRPO training from levels 1–2 to levels 1–4 (ablation) | Qwen2.5-7B-Instruct · Arithmetic GCD, level 5 | accuracy: 3% to 3% | No clear effect |
| Train with GRPO on individual skills, without training on their composition | Qwen2.5-7B-Instruct · Compositional OOD tasks | accuracy: only minor gains (e.g., +6%) in most cases | No clear effect |
| Use the original skill pairings instead of replacing one or both skills with nearby alternatives (ablation) | Qwen2.5-7B-Instruct compositional RL · Strongest-performing compositional settings | accuracy gain: +7.5 pp and +15 pp | Helped |
| Replace either component of the original compositional skill pair (ablation) | Qwen2.5-7B-Instruct compositional RL · Strongest-performing compositional settings | accuracy gain | Hurt |
| Replace both components of the original compositional skill pair (ablation) | Qwen2.5-7B-Instruct compositional RL · Strongest-performing compositional settings | accuracy gain | Hurt |
| Apply RL on familiar strategies | Qwen2.5-7B-Instruct · Transformative OOD tasks | accuracy: often 0% | No clear effect |
| Apply RL on conventional matrix-rank problems | Qwen2.5-7B-Instruct · Transformative Setting 1, matrix rank OOD | accuracy: 70%; dropping 30 percentage points | Hurt |
| Increase inference-time candidate count from 1 to 32 | Advanced LLMs · Letter distribution problems, increasing complexity | Pass@k: gradually approaching 100% when the problem complexity is low; performance drops to zero at complexity level 6 | Mixed |
Surprising findings
- On matrix rank, RL training reduced transformative OOD accuracy by about 30 points from a base model that initially achieved about 70%.
- Lower accuracy at high complexity stems partly from models reducing explicit computation in favor of conjectural guesses, not only from arithmetic errors.
Limitations
- Problems are template-generated and restricted to single-scope families, which may not reflect natural competition math.
- Compositional and transformative settings contain only seven categories each, limiting coverage.
- Pass@k was only tested up to 64 attempts due to budget constraints.
Experimental setup
Datasets: OMEGA templated problem generators
Benchmarks: OMEGA
Metrics: exact-match accuracy pass@k
Baselines: DeepSeek-R1 Claude-3.7-Sonnet OpenAI-o3-mini OpenAI-o4-mini Qwen2.5-Math-7B
Builds on: GRPO Boden's typology of creativity GSM-Symbolic GSM-Infinite
Closest papers by meaning
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.90)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.90)
- Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities (NeurIPS 2025 · similarity 0.90)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.90)
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.