Research map / Architectures and efficiency / Pruning and efficient tuning
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
Dami Choi, Derrick Xin, Hamid Dadkhahi, Justin Gilmer, Ankush Garg, Orhan Firat, Chih-Kuan Yeh, Andrew M. Dai, Behrooz Ghorbani
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on En->Fr then jointly fine-tune on En->Ro + En->Fr | scalarization · En->{Ro,Fr} NMT | low-resource (En->Ro) loss trade-off front | Improved |
| pre-train on high-resource pair then jointly fine-tune on En->Hi + En->Fr | scalarization · En->{Hi,Fr} NMT | En->Hi loss trade-off front | Improved |
| pre-train on En->Fr before training En->Zh + En->Fr (both high-resource) | scalarization · En->{Zh,Fr} NMT | loss trade-off front | No clear effect |
| add optimizer and LR-schedule reset to scalarization (restart baseline) (ablation) | scalarization · En->{Ro,Fr} NMT | En->Ro loss | No clear effect |
| fine-tune only on low-resource task after pre-training (classic transfer) instead of joint fine-tuning (ablation) | pre-training joint fine-tuning · En->{Ro,Fr} NMT | loss | Hurt |
| increase regularization strength (dropout) in scalarization instead of pre-training (ablation) | scalarization · En->{Ro,Fr} NMT | En->Ro loss | No clear effect |
| pre-train English then jointly fine-tune 5 languages | mT5-XXL temperature sampling (tau=3.33) · mC4 Gu, Gd, Sw | language modeling loss | Improved |
| pre-train English then jointly fine-tune 5 languages | mT5-XXL temperature sampling (tau=3.33) · mC4 English | language modeling loss | Improved |
| add learning-rate schedule restart to static sampling (ablation) | mT5-XXL static sampling · mC4 | language modeling loss | Improved |
| lengthen pre-training under fixed 500k total steps (ablation) | mT5-XL · mC4 Gu, Gd, Sw | language modeling loss | Improved |
| lengthen pre-training under fixed 500k total steps (ablation) | mT5-XL · mC4 English and Hindi | language modeling loss | Hurt |
| lengthen pre-training on En->Fr with unfixed budget (ablation) | NMT pre-training joint fine-tuning · En->{Ro,Fr} NMT | overall trade-off front | Improved |
Closest papers by meaning
- Language Imbalance Driven Rewarding for Multilingual Self-improving (ICLR 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.87)
- Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.86)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.86)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.86)
- PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
