Research map / Label-efficient and robust learning / Knowledge distillation
Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAK, a multi-teacher distillation framework that preserves each vision foundation model's representation bias via teacher-specific adapter paths and a shared stem, then adaptively fuses them with mixture-of-representations routers for multi-task dense prediction.
Problem
Multi-teacher distillation of vision foundation models into one student erases the distinct representation biases of each teacher, causing suboptimal multi-task dense prediction performance.
Approach
A Teacher-Agnostic Stem shared across all teachers plus lightweight Teacher-Specific Adapter Paths that mirror each teacher's representation, with task-specific Mixture-of-Representations routers that compute global (image-level) gating over the shared and teacher-specific experts.
Key results
- On NYUD-v2 with ViT-L backbones, SAK reaches MTL Gain of 14.05% versus 4.46% for the best prior MTL model (BFCI) and 8.95% for RADIO.
- On PASCAL-Context with ViT-L backbones, SAK achieves Delta_m of 2.30% versus 0.29% for RADIO, with Semseg 84.01 and Parsing 76.99 mIoU.
- On NYUD-v2 with ViT-B backbones, SAK reaches 11.11% MTL Gain versus 6.33% for RADIO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAK instead of RADIO | ViT-B multi-task model · NYUD-v2 | MTL Gain: 6.33 -> 11.11 | Helped |
| Use SAK instead of the strongest prior method | ViT-B multi-task model · PASCAL-Context | MTL Gain: -0.83 -> 0.83 | Helped |
| Add Teacher-Specific Adapter Paths to preserve teacher representation biases (ablation) | SAK · PASCAL-Context | representation similarity | Helped |
| Add Mixture-of-Representations Routers to fuse biased representations adaptively (ablation) | SAK · PASCAL-Context | multi-task performance | Helped |
| Replace MoR Routers with simple addition of biased representations (ablation) | SAK with TSAP · PASCAL-Context | multi-task performance | Hurt |
| Include Stage 1 ImageNet teacher distillation (ablation) | SAK training · PASCAL-Context | multi-task performance | Helped |
| Include distillation loss during Stage 2 (ablation) | SAK training · PASCAL-Context | final outcomes | Helped |
Surprising findings
- Naive many-to-one distillation (student without biases) underperforms the best single teacher by over 6% on Parsing and Boundary tasks on PASCAL-Context.
- Frozen SAM underperforms the plain ImageNet ViT baseline on semantic segmentation by about 30%.
Limitations
- Experiments are limited to two multi-task benchmarks (PASCAL-Context and NYUD-v2) with a simple decoder head design.
- Stage 1 distillation uses only ImageNet-1k to keep fairness with ViT baselines, which may leave performance on the table versus larger distillation data.
Experimental setup
Datasets: ImageNet-1k PASCAL-Context NYUD-v2
Benchmarks: PASCAL-Context NYUD-v2
Metrics: mIoU maxF mErr odsF RMSE MTL Delta_m
Baselines: RADIO Theia InvPT InvPT++ TaskPrompter TaskExpert BFCI MLoRE TSP
Builds on: RADIO Theia Mixture-of-Experts MLoRE
Closest papers by meaning
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.89)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.88)
- AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models (CVPR 2026 · similarity 0.88)
- Hierarchical Knowledge Prompt Tuning for Multi-task Test-Time Adaptation (CVPR 2025 · similarity 0.87)
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.