Research map / Label-efficient and robust learning / Recognition and long-tail
Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes distribution matching and soft co-annotation losses that couple classification tasks via task-relatedness (from cognitive-study or empirical annotations) to enable multi-task learning with little or non-overlapping labels and avoid negative transfer.
Problem
Multi-task learning typically requires fully or largely overlapping annotations across tasks; with sparse, non-overlapping labels or large label-size imbalance, MTL suffers from negative transfer.
Approach
Encode task relatedness (e.g., expression-AU prototypical/observational links) as a deterministic mapping; match AU predictions to a distribution induced by expression predictions (distribution matching loss) and convert AU labels into soft expression targets (soft co-annotation loss), added to standard MTL losses without changing architecture or inference.
Key results
- Coupling losses improved each multi-task SOTA network by roughly 5-6% over its non-coupled MT version on expression and AU tasks across databases, eliminating negative transfer observed in NC MT.
- On CAD with VGG/ResNet/DenseNet, coupled MT models outperformed single-task counterparts by 10-14% total accuracy for classification and 4-5% for attributes.
- On TMIF-FEA (Aff-Wild2), using both losses with Aff-Wild2-inferred relatedness raised expression F1 from 0.337 to 0.399 and AU F1 from 0.52 to 0.578.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add both soft co-annotation and distribution matching (ablation) | TMIF-FEA · Aff-Wild2 | F1 (expression): 0.337 -> 0.399 | Helped |
| Add both soft co-annotation and distribution matching (ablation) | TMIF-FEA · Aff-Wild2 | F1 (AU): 0.52 -> 0.578 | Helped |
| Use soft co-annotation only (ablation) | TMIF-FEA · Aff-Wild2, DM relatedness | F1 (expression): 0.337 -> 0.377 | Helped |
| Use soft co-annotation only (ablation) | TMIF-FEA · Aff-Wild2, DM relatedness | F1 (AU): 0.52 -> 0.544 | Helped |
| Use distribution matching only (ablation) | TMIF-FEA · Aff-Wild2, DM relatedness | F1 (expression): 0.337 -> 0.374 | Helped |
| Use distribution matching only (ablation) | TMIF-FEA · Aff-Wild2, DM relatedness | F1 (AU): 0.52 -> 0.547 | Helped |
| Use soft co-annotation only (ablation) | TMIF-FEA · Aff-Wild2, Aff-Wild2 relatedness | F1 (expression): 0.337 -> 0.388 | Helped |
| Use distribution matching only (ablation) | TMIF-FEA · Aff-Wild2, Aff-Wild2 relatedness | F1 (AU): 0.52 -> 0.565 | Helped |
| Train with MTL without coupling instead of single-task models | Affective computing networks · AffectNet, RAF-DB, ABAW4 LSD | expression recognition | Hurt |
| Add the proposed coupling losses to MTL | Affective computing networks · AffectNet, RAF-DB, ABAW4 LSD, Aff-Wild2 and EmotioNet | task performance: approximately: i) 5% on Affectnet and 5% on EmotioNet; ii) 6% on RAF-DB and 5% on EmotioNet; iii) 5% on ABAW4 LSD and 6% | Helped |
| Add coupling losses to MTL | VGG-16, ResNet-50 and DenseNet-121 · CelebA, CUB, S-ADD and CAD | classification and attribute performance | Helped |
Surprising findings
- Naive multi-task training without coupling caused negative transfer for basic expression recognition, even though AU detection improved.
- Student-teacher knowledge distillation for building complete labels gave only slight gains and still exhibited negative transfer for expressions.
Limitations
- Requires a direct, known or empirically inferable relationship between tasks; method is not applicable otherwise.
- Relatedness mapping is deterministic and derived from a limited cognitive study or a single dataset's annotations.
Experimental setup
Datasets: AffectNet RAF-DB ABAW4 LSD Aff-Wild2 EmotioNet CelebA CUB-200-2011 Shoes (S-ADD) Clothing Attributes Dataset (CAD)
Benchmarks: AffectNet RAF-DB ABAW4 LSD Aff-Wild2 EmotioNet CelebA CUB-200-2011 Shoes (S-ADD) Clothing Attributes Dataset (CAD)
Metrics: accuracy F1 average accuracy AFA (average of F1 and accuracy)
Baselines: MGDA-UB GradNorm AdaShare TR Student-Teacher knowledge distillation EmoAffectNet EffNet-B2 PSR VGGFACE MTER-KDTD HSE-NN TMIF-FEA
Builds on: GradNorm MGDA AdaShare NTS-Net knowledge distillation
Closest papers by meaning
- MTL-UE: Learning to Learn Nothing for Multi-Task Learning (ICML 2025 · similarity 0.87)
- StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets (CVPR 2026 · similarity 0.87)
- Towards Principled Task Grouping for Multi-Task Learning (NeurIPS 2026 · similarity 0.86)
- Identifying and Mitigating Spurious Correlation in Multi-Task Learning (CVPR 2025 · similarity 0.86)
- CocoER: Aligning Multi-Level Feature by Competition and Coordination for Emotion Recognition (CVPR 2025 · similarity 0.86)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.