Research map / Label-efficient and robust learning / Recognition and long-tail
An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoMTL, an information-theoretic multi-task learning framework that maximizes shared information (SIMax) and minimizes task-specific information (TIMin) to learn noise-invariant sufficient representations for PLM-based text classification.
Problem
Multi-task learning on noisy, limited-label social media NLU data: information-bottleneck MTL methods compress shared representations and cause task-specific insufficiency, and task-balancing methods perform suboptimally under noise and data scarcity.
Approach
Instead of compressing shared representations as IB-based MTL does, maximize shared-representation information about the input (via an InfoNCE lower bound on I(X;Z)) and the targets (cross-entropy), while compressing each task's output representation with a variational Gaussian bottleneck (KL term) to remove task-irrelevant redundancy.
Key results
- With RoBERTa, InfoMTL reaches Avg 68.50 and Δp +3.97% over the EW baseline (66.17), the best of 12 MTL methods compared.
- With BERT, InfoMTL reaches Avg 67.51 and Δp +3.70%, versus EW at 65.62.
- Under noisy perturbations, InfoMTL improves robust scores by +29.2% (random) and +27.8% (adversarial) relative to EW.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use InfoMTL instead of EW | BERT multi-task learning · Six benchmarks (average) | Avg.: 65.62 -> 67.51 | Helped |
| use InfoMTL instead of EW | RoBERTa multi-task learning · Six benchmarks (average) | Avg.: 66.17 -> 68.50 | Helped |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · HatEval | M-F1: 48.52 -> 48.44 | Hurt |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · EmotionEval | M-F1: 74.74 -> 76.90 | Helped |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · IronyEval | F1(i.): 68.55 -> 68.94 | Helped |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · OffensEval | M-F1: 79.04 -> 79.78 | Helped |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · SentiEval | M-Recall: 71.45 -> 71.92 | No clear effect |
| use InfoMTL instead of the strongest reported comparative method | RoBERTa multi-task learning · StanceEval | M-F1 (a. & f.): 63.76 -> 65.02 | Helped |
| add SIMax (ablation) | InfoMTL with BERT · Six benchmarks (average) | Avg.: 66.01 -> 67.51 | Helped |
| add TIMin (ablation) | InfoMTL with BERT · Six benchmarks (average) | Avg.: 67.02 -> 67.51 | Helped |
| add SIMax (ablation) | InfoMTL with RoBERTa · Six benchmarks (average) | Avg.: 67.59 -> 68.50 | Helped |
| add TIMin (ablation) | InfoMTL with RoBERTa · Six benchmarks (average) | Avg.: 67.55 -> 68.50 | Helped |
Surprising findings
- Zero-shot GPT-3.5 is reported to score 55.32 average, well below the fine-tuned RoBERTa models.
- Several existing MTL methods (e.g., DWA and PCGrad with BERT, IMTL-L) fall below the EW baseline on average.
Limitations
- Evaluated only on six English tweet classification benchmarks with BERT and RoBERTa backbones.
- Hyperparameters (alpha, beta, tau) are tuned per backbone via search on validation sets, and gains are modest (about +2 points average).
Experimental setup
Datasets: EmotionEval HatEval IronyEval OffensEval SentiEval StanceEval
Benchmarks: EmotionEval HatEval IronyEval OffensEval SentiEval StanceEval
Metrics: macro-F1 F1 (ironic class) macro-recall Avg. relative improvement Δp ARI uniformity robust score
Baselines: EW TW SI UW GLS DWA PCGrad IMTL-L RLW MT-VIB VMTL Hierarchical MTL
Builds on: information bottleneck variational information bottleneck InfoNCE uncertainty weighting MT-VIB
Closest papers by meaning
- A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks (AAAI 2019 · similarity 0.88)
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (ICLR 2018 · similarity 0.87)
- A Mutual Information Maximization Perspective of Language Representation Learning (ICLR 2020 · similarity 0.86)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.84)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.84)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.