Research map / Label-efficient and robust learning / Self-supervised and contrastive
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with C4 span denoising and ExMix | T5.1.1 base · SuperGLUE validation | AVG: 76.1 -> 79.9 | Helped |
| Pre-train with C4 span denoising and ExMix | T5.1.1 large · SuperGLUE validation | AVG: 85.3 -> 87.3 | Helped |
| Pre-train with C4 span denoising and ExMix | T5.1.1 large · Rainbow validation | AVG: 83.42 -> 84.07 | Helped |
| Pre-train with C4 span denoising and ExMix | T5 large · MSMarco | MRR@10: 0.393 -> 0.402 | Helped |
| Use all 107 ExMix tasks instead of a transfer-selected subset | T5 base pre-training · SuperGLUE validation | AVG: 76.4 -> 79.9 | Helped |
| Pre-train on ExMix rather than adding it as a separate pre-finetuning stage (ablation) | T5 base · SuperGLUE validation | AVG: 78.1 -> 79.9 | Helped |
| Add C4 span-denoising examples to ExMix pre-training (ablation) | ExT5 · SuperGLUE validation | AVG: 65.0 -> 79.9 | Helped |
| Set the C4-to-ExMix example ratio to 2 (ablation) | ExT5 base · SuperGLUE validation | AVG: 77.3 -> 79.9 | Helped |
| Increase the C4-to-ExMix ratio from 2 to 4 (ablation) | ExT5 base · SuperGLUE validation | AVG: 79.9 -> 77.6 | Hurt |
| Increase the C4-to-ExMix ratio from 2 to 5 (ablation) | ExT5 base · SuperGLUE validation | AVG: 79.9 -> 76.2 | Hurt |
| Increase the number of pre-training tasks from 30 to 107 (ablation) | ExT5 with batch size 512 · SuperGLUE validation | AVG: 75.8 -> 79.9 | Helped |
| Increase the number of pre-training tasks from 30 to 107 (ablation) | ExT5 with batch size 128 · SuperGLUE validation | AVG: 74.1 -> 75.6 | Mixed |
Closest papers by meaning
- On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion (NeurIPS 2024 · similarity 0.86)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.86)
- MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse Upcycling (ICLR 2025 · similarity 0.86)
- Zero-Shot Performance Prediction for Probabilistic Scaling Laws (NeurIPS 2025 · similarity 0.86)
- PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts (NeurIPS 2025 · similarity 0.86)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.