Research map / Learning theory and algorithms / Prediction and decision losses
A Theoretical Framework for Modular Learning of Robust Generative Models
Corinna Cortes, Mehryar Mohri, Yutao Zhong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a frozen-expert gate instead of retraining one monolithic model on the union of domains | ~20M-param modular model · Wiki/Code/FineWeb NLL (diff seed) | NLL per token: 5.133 -> 4.994 | Improved |
| use a frozen-expert gate instead of retraining on the union (different data splits) | ~20M-param modular model · Wiki/Code/FineWeb NLL (diff data) | NLL per token: 5.306 -> 5.087 | Improved |
| use a frozen-expert gate instead of a jointly trained token-level MoE with load balancing | ~20M-param modular model · Wiki/Code/FineWeb NLL (diff seed) | NLL per token: 5.080 -> 4.994 | Improved |
| scale to 1.6B-parameter experts with a 2.5M-param gate versus a 4.9B monolithic model | Robust Gate · real-world data, 1B+ scale | test NLL: 4.829 -> 4.734 | Improved |
| use gate rather than fixed-mixture retrained models, including larger ones | Robust Gate · synthetic contradictory domains (+1/-1 rules) | NLL per token | Improved |
| compare gate with oracle models retrained on exact test mixture | Robust Gate · synthetic task | NLL per token | Mixed |
| distill non-causal gate into a causal router (Structural Distillation) | Causal Router (d=10) · synthetic task | NLL per token | Improved |
| increase monolithic baseline capacity to 1.5x (ablation) | Retrained Fixed model · synthetic contradictory domains | NLL per token | No clear effect |
| use Lagrangian primal-dual instead of quadratic penalty for normalization (ablation) | Gate training · algorithm design | constraint satisfaction | Improved |
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.86)
- Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.86)
- Score-based generative models are provably robust: an uncertainty quantification perspective (NeurIPS 2024 · similarity 0.86)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.85)
- Amortizing intractable inference in large language models (ICLR 2024 · similarity 0.85)
- A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
