Research map / Label-efficient and robust learning / Recognition and long-tail
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
Lucio M. Dery, Paul Michel, Ameet Talwalkar, Graham Neubig
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the end-task objective through multi-task training with TAPT MLM | RoBERTa_base continued pre-training · ACL-ARC, SCIERC, and CHEMPROT | F1: ACL-ARC: 67.74_{3.68} -> 70.48_{4.42}; SCIERC: 79.53_{1.93} -> 80.81_{0.74}; CHEMPROT: 82.17_{0.65} -> 84.29_{0.63} | Improved |
| meta-learn task weights for end-task and TAPT MLM | TAPT continued pre-training · ACL-ARC, SCIERC, and CHEMPROT | F1: ACL-ARC: 67.74_{3.68} -> 70.08_{4.70}; SCIERC: 79.53_{1.93} -> 81.48_{0.82}; CHEMPROT: 82.17_{0.65} -> 84.49_{0.50} | Improved |
| add the end-task objective to DAPT and TAPT through multi-task training | DAPT+TAPT · SCIERC and CHEMPROT, n = 10 × |Train| | F1: SCIERC: 77.62_{1.38} -> 81.02_{1.24}; CHEMPROT: 78.22_{0.74} -> 83.77_{0.60} | Improved |
| add the end-task objective to DAPT and TAPT, using fixed equal task weights | DAPT+TAPT · ACL-ARC, n = 10 × |Train| | macro-F1: 69.12_{5.76} -> 71.58_{1.65}; p = 0.110 | Mixed |
| use META-TARTAN instead of MT-TARTAN to adapt weights for domain-only DAPT | DAPT-only TARTAN · ACL-ARC, SCIERC, and CHEMPROT | F1: ACL-ARC: 69.27_{0.96} -> 71.19_{4.88}; SCIERC: 81.53_{0.99} -> 82.08_{1.19}; CHEMPROT: 80.26_{3.79} -> 82.31_{0.75} | Improved |
| use a separate meta-classification head instead of the end-task training head (ablation) | META-TARTAN with ACL-ARC and DAPT · ACL-ARC | F1: a delta of 1.7 F1 points | Improved |
| use a separate meta-head rather than the same end-task head for calculating meta-gradients (ablation) | META-TARTAN · ACL-ARC with DAPT | auxiliary task weight utilization | Improved |
| use task-aware multi-task or meta-learning instead of task-agnostic pre-training | CIFAR100 Medium-Sized Mammals classification · Medium-Sized Mammals | accuracy: 46.7_{2.2} -> 51.3_{1.2}; 46.7_{2.2} -> 52.3_{3.8} | Improved |
| use META-TARTAN instead of TAPT | HYPER-PARTISAN classification · HYPER-PARTISAN | accuracy: 93.39_{2.26} -> 96.84_{1.72} | Improved |
| use META-TARTAN with 10 × |Train| DAPT data instead of full-data DAPT | ACL-ARC · ACL-ARC | macro-F1: 71.19_{4.88} vs 72.49_{3.28} | Mixed |
| use META-TARTAN with 10 × |Train| auxiliary data instead of end-task-agnostic methods using full domain data | SCIERC · SCIERC | macro-F1: 81.41_{1.70} vs 79.97_{2.11}; 81.41_{1.70} vs 80.00_{1.08} | Improved |
| use META-TARTAN with 10 × |Train| auxiliary data instead of full-data DAPT or DAPT+TAPT | CHEMPROT · CHEMPROT | micro-F1: 83.38_{0.89} vs 86.54_{1.05}; 83.38_{0.89} vs 87.24_{0.81} | Hurt |
Closest papers by meaning
- AANG : Automating Auxiliary Learning (ICLR 2023 · similarity 0.90)
- Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (ICLR 2021 · similarity 0.89)
- Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner (NeurIPS 2023 · similarity 0.88)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.86)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.86)
- Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
