Research map / Large language models / Reasoning and chain of thought

Learning from Synthetic Data Improves Multi-hop Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RL fine-tuning small LLMs on rule-generated synthetic multi-hop reasoning data (fictional worlds) improves real-world multi-hop QA benchmarks, with gains attributed to knowledge composition rather than memorization or formatting.

Problem

RLVR needs abundant verifiable data, which is scarce for human annotation and costly or unreliable with LLM-generated data or LLM verifiers. It is unclear whether rule-generated synthetic data with fictional entities can teach reasoning skills that transfer to real-world multi-hop QA.

Approach

Use template/grammar/logic-program generated fictional-world datasets (PhantomWiki, GSM-Infinite, ReasoningGym Family-Relationships and Knights-Knaves) as RLVR training data with GRPO, and show that outcome-only RL on these zero-overlap datasets transfers to real multi-hop QA, outperforming SFT on the same synthetic data.

Key results

• Qwen3-0.6B RL-fine-tuned on PhantomWiki achieves relative F1 gains of 56% to 131% on real-world multi-hop benchmarks (per paper introduction).
• On Qwen3-0.6B, SFT on GSM-Infinite raises HotpotQA F1 from 0.357 to 0.400 at best, versus 0.479 with RL on the same synthetic setting; SFT does not improve real-world performance as RL does for Qwen3-1.7B (0.594 base, 0.576 SFT, 0.666 RL).
• Format-only RL raises Qwen2.5-1.5B-Instruct HotpotQA F1 from 0.02 to 0.43, but does not help Qwen3 or Phi-4-mini-reasoning, indicating the Qwen3 and Phi gains come from knowledge composition.

Surprising findings

• Format-only binary reward lifts Qwen2.5-1.5B-Instruct substantially on real-world benchmarks, revealing reward hacking through formatting.
• SFT on synthetic solution traces improves the synthetic task but yields no real-world gain, while RL transfers.

Limitations

• Larger models (Qwen3-4B, Qwen2.5-7B-Instruct) show less consistent transfer, possibly due to suboptimal reduced hyperparameters forced by memory limits.
• Evaluation uses distractor settings with in-context evidence, not open-domain retrieval.
• Transfer extent to other reasoning skills and mixed memorization/composition effects remain unclear.

Experimental setup

Datasets: PhantomWiki GSM-Infinite ReasoningGym Family-Relationships ReasoningGym Knights-Knaves

Benchmarks: HotpotQA 2WikiMultihopQA MuSiQue CofCA SynthWorlds-RM GSM-Infinite

Metrics: F1 accuracy standard error

Baselines: SFT on GSM-Infinite solution traces format-only RL reward RL on in-domain real-world training data

Builds on: GRPO PhantomWiki GSM-Infinite ReasoningGym TRL

Closest papers by meaning

• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.89)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.89)
• Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers (ICML 2025 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.89)
• RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.89)