Research map / Optimization / Optimizers
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GradVac with layer-wise, EMA-based gradient similarity objectives | Multilingual MT · WMT En→Any (avg) | BLEU: 22.48 -> 23.89 | Helped |
| Use GradVac with layer-wise, EMA-based gradient similarity objectives | Multilingual MT · WMT Any→En (avg) | BLEU: 25.61 -> 26.34 | Helped |
| Use GradVac | Massive multilingual MT · Any→En (All) | BLEU: 26.71 -> 27.21 | Helped |
| Use GradVac | Massive multilingual MT · En→Any (All) | BLEU: 18.25 -> 19.08 | Helped |
| Use GradVac for joint finetuning | mBERT · XTREME NER (avg) | F1: 82.9 -> 83.8 | Helped |
| Replace EMA task-specific objectives with a fixed gradient similarity objective (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 22.99 | Hurt |
| Apply GradVac to the whole model rather than independently per layer (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.30 | Hurt |
| Apply GradVac to high-resource tasks only instead of low-resource tasks only (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.78 | Hurt |
| Apply GradVac to all tasks instead of low-resource tasks only (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.64 | Hurt |
| Use encoder-decoder granularity instead of per-layer granularity (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.66 | Hurt |
| Use per-matrix granularity instead of per-layer granularity (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.52 | Hurt |
| Set EMA decay β to 1e-1 instead of 1e-2 (ablation) | GradVac · WMT En→Any (avg) | BLEU: 23.89 -> 23.31 | Hurt |
Closest papers by meaning
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.88)
- Objective Soups: Multilingual Multi-Task Modeling for Speech Processing (NeurIPS 2025 · similarity 0.88)
- Model Fusion through Bayesian Optimization in Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.