Research map / Large language models / Agents, code and math
TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TreeSynth, which recursively partitions a task's data space into a tree of mutually exclusive, exhaustive subspaces with LLMs and synthesizes samples per leaf, yielding more diverse synthetic data for math, code and psychology tasks.
Problem
LLM-based data synthesis from scratch suffers from low diversity, repetition and biased distributions as data scale grows, due to model biases, limited seed data and low-variation prompts.
Approach
Treat the task data space as the root of a decision-tree-like spatial partitioning tree: LLMs pick a criterion and mutually exclusive, exhaustive attribute values at each node, and samples are generated within each leaf subspace; the same tree can route and rebalance existing datasets.
Key results
- With GPT-4o-powered synthesis, Qwen2.5-7B trained on TreeSynth data averages 74.06 vs 61.82 for vanilla data (Avg. across GSM8K, MATH, MBPP, HumanEval).
- TreeSynth data has lower cosine-similarity diversity scores than vanilla and peer methods in most settings (e.g., 0.36 GSM8K vs 0.40 vanilla for LLaMA3.3-70B generation).
- On SimpleToM, TreeSynth-guided balancing improves Temperature Sampling accuracy by 7.9% and Persona Hub by 2.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TreeSynth-generated data instead of Persona Hub data | Qwen2.5-7B · GSM8K | accuracy: 83.24 -> 86.13 | Helped |
| Use TreeSynth-generated data instead of Persona Hub data | Qwen2.5-7B · MATH | accuracy: 66.22 -> 66.84 | Helped |
| Use TreeSynth-generated data instead of Persona Hub data | Qwen2.5-7B · MBPP | accuracy: 61.6 -> 62.8 | Helped |
| Use TreeSynth-generated data instead of Temperature Sampling data | Qwen2.5-7B · HumanEval | accuracy: 80.49 -> 80.49 | No clear effect |
| Use TreeSynth-generated data instead of Persona Hub data | LLaMA3.1-8B · SimpleToM | accuracy: 85.7 -> 88.0 | Helped |
| Apply TreeSynth-guided data balancing (ablation) | Temperature Sampling data, trained on LLaMA3.1-8B · SimpleToM | accuracy: 78.9 -> 86.8 | Helped |
| Apply TreeSynth-guided data balancing (ablation) | Persona Hub data, trained on LLaMA3.1-8B · SimpleToM | accuracy: 85.7 -> 88.6 | Helped |
| Apply TreeSynth-guided data balancing (ablation) | Persona Hub data · SimpleToM | diversity (cosine similarity): 0.33 -> 0.32 | Helped |
Surprising findings
- Temperature Sampling and Persona Hub reach saturation or degrade at larger data scales on several benchmarks, and Evol-Instruct and Persona Hub underperform vanilla data on HumanEval.
Limitations
- Fine-tuning hyperparameters were selected by a broad grid search, and gains depend on configuration choices.
- Diversity is measured only by cosine similarity of embeddings, a limited proxy for coverage.
- Tree construction relies on LLM-proposed criteria and attribute values without human verification.
Experimental setup
Datasets: GSM8K training set MATH training set Code Alpaca (Python subset) SimpleToM
Benchmarks: GSM8K MATH MBPP HumanEval SimpleToM
Metrics: accuracy pairwise cosine similarity diversity (lower is more diverse)
Baselines: Vanilla Data Temperature Sampling Evol-Instruct Persona Hub Zero-Shot Few-Shot
Builds on: decision trees Evol-Instruct Persona Hub LoRA
Closest papers by meaning
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.85)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.85)
- Diffusion Tree Sampling: Scalable inference‑time alignment of diffusion models (NeurIPS 2025 · similarity 0.84)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.84)
- The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph (ICML 2025 · similarity 0.84)
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.