Research map / Label-efficient and robust learning / Recognition and long-tail
Learning Multi-Task Sparse Representation Based on Fisher Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSMTL, a deep multi-task learning method that learns per-task binary sparse masks over shared parameters by selecting the top-K parameters ranked by an empirical Fisher information estimate, alternating mask and weight updates.
Problem
Negative knowledge transfer in deep multi-task learning when tasks are weakly correlated or features conflict, since all tasks share the full feature extractor.
Approach
Replace random or evolutionary sparse sharing with a Fisher-information-based sparse module: each task keeps the K shared parameters with the largest (empirically estimated) Fisher information, and network weights and task masks are updated alternately, so only jointly optimized parameters transfer knowledge.
Key results
- On CelebA, FS reaches 0.910 accuracy and 0.681 F-score using only 10% of parameters, versus 0.894 accuracy for MTL and 0.899 for SE.
- On Cityscapes, FS with 30% parameters achieves 66.24 mIoU versus 56.57 for MTL and 58.57 for STL.
- On DKL-mnist, FS with 90% kept parameters achieves 0.880 accuracy, slightly above STL (0.877) and above MTL (0.869).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Fisher-based sparse sharing instead of single-task learning | FSMTL · DKL-mnist | Accuracy: 0.877 ± 0.002 -> 0.880 ± 0.001 | Helped |
| Use Fisher-based sparse sharing instead of single-task learning | FSMTL · DKL-mnist | Precision: 0.807 ± 0.006 -> 0.811 ± 0.006 | Helped |
| Use Fisher-based sparse sharing instead of single-task learning | FSMTL · DKL-mnist | Recall: 0.807 ± 0.005 -> 0.813 ± 0.007 | Helped |
| Use Fisher-based sparse sharing instead of single-task learning | FSMTL · DKL-mnist | F-score: 0.797 ± 0.006 -> 0.801 ± 0.007 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · CelebA | Accuracy: 0.904 ± 0.001 -> 0.910 ± 0.0003 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · CelebA | Precision: 0.728 ± 0.004 -> 0.753 ± 0.002 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · CelebA | Recall: 0.626 ± 0.005 -> 0.637 ± 0.003 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · CelebA | F-score: 0.659 ± 0.002 -> 0.681 ± 0.0004 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · Cityscape | mIoU: 60.50 ± 0.19 -> 66.24 ± 0.34 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · Cityscape | Pix. Acc.: 97.53 ± 0.05 -> 98.05 ± 0.02 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · Cityscape | Abs. Err.: 0.0130 ± 0.0001 -> 0.0125 ± 0.0001 | Helped |
| Use Fisher-based sparse sharing instead of the strongest baseline | FSMTL · Cityscape | Rel. Err.: 25.09 ± 0.28 -> 25.86 ± 0.647 | Hurt |
Surprising findings
- Standard MTL performs worse than single-task learning on DKL-mnist and Cityscapes, indicating negative transfer.
Limitations
- Analytical experiments restricted to the smaller DKL-mnist and CelebA datasets due to time cost.
- Gains on heterogeneous-feature tasks (DKL-mnist) are smaller than on homogeneous tasks, per the authors.
Experimental setup
Datasets: DKL-mnist CelebA Cityscapes
Benchmarks: DKL-mnist CelebA Cityscapes
Metrics: Accuracy Precision Recall F-score mIoU Pixel Accuracy Abs. Err. Rel. Err.
Baselines: STL MTL GradNorm MGDA-UB CAGrad RLW Nash-MTL SE TR MR RS EFS
Builds on: sparse sharing (Sun et al. 2020a) natural gradient Fisher approximation Random Loss Weighting
Closest papers by meaning
- Learning Sparse Sharing Architectures for Multiple Tasks (AAAI 2020 · similarity 0.88)
- Learning Sparse Task Relations in Multi-Task Learning (AAAI 2017 · similarity 0.86)
- Improved Active Multi-Task Representation Learning via Lasso (ICML 2023 · similarity 0.84)
- Deep Multi-task Representation Learning: A Tensor Factorisation Approach (ICLR 2017 · similarity 0.84)
- Near-Optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation (ICML 2026 · similarity 0.83)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.