Research map / Large language models / Language, safety and interpretability

Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretical analysis of synthetic data in LLM post-training via a distributional model of generation, a "reverse-bottleneck" information-gain view, and a GGMI metric, validated with GMM simulations and a small Dolly-15K experiment.

Problem

Lack of a rigorous theoretical understanding of why synthetic data generated by LLMs improves post-training generalization, and how generation factors (information gain, prompt, noise) affect it.

Approach

Models synthetic generation as a Markov chain from anchor data to prompt to synthetic data, introduces information gain from the generator M as a "reverse bottleneck" relative to classic IB, derives generalization upper bounds in terms of this gain, and defines GGMI to compare synthetic versus anchor-only post-training.

Key results

• KL Gap increases with J (unsampled target components) and decreases with K (anchor components) and L (task-irrelevant components) in GMM simulation, consistent with the GGMI bound.
• Authors state they could not validate findings in real-world LLM settings due to compute limits; the LLM experiment is only partially described in the appendix.

Measured effects reported in the paper

Surprising findings

• Adding task-irrelevant components to the generator (larger L) reduces the KL gap, showing a trade-off in generator choice.

Limitations

• Validation limited to 2D GMM simulations; real LLM experiments not conducted due to compute limits.
• Several bounds rely on strong assumptions (sub-Gaussian loss, contraction layers, reversible prompt transform, alpha-relation of entropies).
• Mutual information terms are hard to estimate, so the bounds are not directly measurable in practice.

Experimental setup

Datasets: Dolly-15K

Benchmarks: GMM simulation (KL Gap)

Metrics: KL divergence gap generalization error upper bound information gain estimate

Builds on: information bottleneck theory mutual information generalization bounds self-instruct

Closest papers by meaning

• It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.90)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
• How to Synthesize Text Data without Model Collapse? (ICML 2025 · similarity 0.89)
• Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.89)
• Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.88)
• Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.88)