Research map / Large language models / Reasoning and chain of thought
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Enigmata, a suite of 36 synthetic verifiable puzzle tasks across 7 categories, an evaluation benchmark, and an RLVR training recipe (RFT followed by multi-task VC-PPO) that improves Qwen2.5-32B on puzzle, ARC-AGI, and OOD tasks.
Problem
LLMs, including strong reasoning models, struggle with knowledge-free logic puzzles, and existing puzzle resources lack diversity, scalability, controllable difficulty, and RLVR-ready verifiers.
Approach
A scalable generator-verifier pair for 36 puzzle tasks with programmatic difficulty control, combined with a recipe of rejection fine-tuning followed by mixed or multi-stage RL (VC-PPO) on puzzles mixed with math and ARC-AGI data.
Key results
- Qwen2.5-32B-Enigmata reaches 62.6 overall on Enigmata-Eval versus 65.1 for o4-mini-high and 12.6 for Qwen2.5-32B-Instruct, and 32.8 on ARC-AGI 1 versus 6.0 for the base model.
- Seed1.5-Thinking-Enigmata improves over Seed1.5-Thinking on AIME 2024 (87.5 vs 86.7), AIME 2025 (75.9 vs 74.0), BeyondAIME (48.4 vs 48.0), and GPQA Diamond (78.1 vs 77.3).
- Balanced difficulty ratio 1:1:1 gives 64.0 on KORBench versus 63.8 for 2:6:2 and 60.1 for HRV, while Mix-Training with SFT-All reaches 62.6 on Enigmata-Eval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rejection fine-tuning before RL | Qwen2.5-32B-Instruct · Enigmata-Eval | accuracy: 12.6 -> 46.6 | Helped |
| Add multi-task RL after rejection fine-tuning | Qwen2.5-32B-RFT · Enigmata-Eval | accuracy: 46.6 -> 62.6 | Helped |
| Add Enigmata puzzle data during RL | Seed1.5-Thinking · AIME 2024 | accuracy: 86.7 -> $87.5_{(+0.8)}$ | Helped |
| Add Enigmata puzzle data during RL | Seed1.5-Thinking · AIME 2025 | accuracy: 74.0 -> $75.9_{(+1.9)}$ | Helped |
| Add Enigmata puzzle data during RL | Seed1.5-Thinking · BeyondAIME | accuracy: 48.0 -> $48.4_{(+0.4)}$ | Helped |
| Add Enigmata puzzle data during RL | Seed1.5-Thinking · GPQA Diamond | accuracy: 77.3 -> $78.1_{(+0.8)}$ | Helped |
| Use a balanced 1:1:1 easy:medium:hard ratio instead of HRV (ablation) | Multi-stage RL · KOR-Bench | accuracy: 60.1 -> 64.0 | Helped |
| Use a balanced 1:1:1 easy:medium:hard ratio instead of medium-focused 2:6:2 (ablation) | Multi-stage RL · AIME 2024 | accuracy: 48.0 -> 56.6 | Helped |
| Use medium-focused 2:6:2 instead of balanced 1:1:1 difficulty sampling (ablation) | Multi-stage RL · ARC-AGI 1 | accuracy: 34.5 -> 35.1 | Helped |
| Increase Enigmata training data in stage 2 (ablation) | Multi-stage RL · Enigmata-Eval | accuracy | Helped |
| Use excessive Enigmata training data in stage 2 (ablation) | Multi-stage RL · OOD benchmarks | performance | Hurt |
| Use Mix Training rather than Multi-Stage RL with SFT-Part (ablation) | Qwen2.5-32B · Enigmata-Eval | accuracy: 56.1 -> 46.7 | Hurt |
Surprising findings
- Qwen2.5-32B shows no clear generalization to math from puzzle RL, while the larger Seed1.5-Thinking model does.
- Using code within responses hurts puzzle accuracy (41.8 with code vs 63.6 without on Enigmata).
Limitations
- Only one backbone (Qwen2.5-32B) and one RL algorithm (VC-PPO) were tested.
- Puzzles are single-turn and text-only; no multi-turn or visual puzzles.
- Six tasks rely on fixed pools from existing datasets rather than auto-generation.
Experimental setup
Datasets: Enigmata-Data ARC-AGI 1 training set ARC-AGI 2 training set AIME 1983-2023 light-R1 math data KOR-Bench
Benchmarks: Enigmata-Eval ARC-AGI 1 ARC-AGI 2 KORBench AIME 2024 AIME 2025 BeyondAIME GPQA Diamond
Metrics: accuracy pass@k
Baselines: o1 o3-mini-high o4-mini-high DeepSeek-R1 Gemini-2.5-Pro Claude-3.7-Sonnet-Thinking QwQ-32B DS-R1-Distilled-Qwen-32B GPT-4o-1120 Grok-2-1212 Seed1.5-Thinking
Builds on: PPO VC-PPO GRPO Light-R1 DeepSeek-R1 VeRL
Closest papers by meaning
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.89)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.89)
- Reinforcing General Reasoning Without Verifiers (ICLR 2026 · similarity 0.89)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)
- LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.89)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.