Research map / Label-efficient and robust learning / Knowledge distillation
Amalgamating Multi-Task Models with Heterogeneous Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VENUS, a knowledge amalgamation method that trains a multi-task student from heterogeneous multi-task teachers on unlabeled data by aligning its shared representation to each teacher's final shared representation via teacher-specific adaptors.
Problem
Training a unified multi-task student from multiple pre-trained multi-task teachers with heterogeneous architectures (different layer structures and representation dimensionalities) using only unlabeled data.
Approach
Introduces a Feature Consolidator with per-teacher trainable adaptors (ReLU(Wx+b)) that project the student's shared representation to each teacher's shared-representation dimensionality, so the student can match teachers' final shared features and their averaged task logits despite architectural heterogeneity.
Key results
- On PASCAL VOC 2007, VENUS achieves average rank 1.85, best among compared methods (Multi-Task CFL 2.55, KD 3.05, MuST 3.45).
- On 3D, VENUS achieves average rank 1.00, with accuracy up to .9784 on purple wall, versus KD at 2.75-3.25 average rank range.
- On CIFAR-10, VENUS achieves average rank 1.60, outperforming Multi-Task CFL (2.70) and KD (2.30).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VENUS instead of Multi-Task CFL | multi-task knowledge amalgamation with heterogeneous teachers · PASCAL VOC 2007 | average rank: 2.55 -> 1.85 | Helped |
| Use VENUS instead of Multi-Task CFL | multi-task knowledge amalgamation with heterogeneous teachers · 3D | average rank: 2.75 -> 1.00 | Helped |
| Use VENUS instead of KD | multi-task knowledge amalgamation with heterogeneous teachers · CIFAR-10 | average rank: 2.30 -> 1.60 | Helped |
| Use VENUS instead of Single-Task CFL | multi-task knowledge amalgamation with DenseNet and VGG teachers · 3D | average rank: 2.75 -> 1.50 | Helped |
| Use VENUS instead of KD | multi-task knowledge amalgamation with DenseNet and AlexNet teachers · 3D | average rank: 2.25 -> 1.75 | Helped |
| Use VENUS instead of Multi-Task CFL | multi-task knowledge amalgamation with VGG and AlexNet teachers · 3D | average rank: 2.50 -> 2.00 | Helped |
Surprising findings
- Multi-task KA baseline MuST performs consistently worst, suggesting pseudo-predictions alone are insufficient for learning high-quality shared features.
- Multi-Task CFL does not show a significantly superior performance over plain KD, though VENUS does.
Limitations
- Evaluated only on binary-classification-style tasks from three image datasets with two teachers in most experiments.
- Requires the student to be trained with teacher representations; adaptors used only during training and must be removed afterward.
Experimental setup
Datasets: PASCAL VOC 2007 3D (3d-shapes) CIFAR-10
Benchmarks: PASCAL VOC 2007 3D CIFAR-10
Metrics: accuracy average rank across tasks
Baselines: Teachers (as is) Single-Task CFL MuST KD Multi-Task CFL
Builds on: Knowledge Amalgamation Knowledge Distillation CFL (common feature learning) MuST
Closest papers by meaning
- Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.89)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.87)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.86)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.86)
- MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · similarity 0.86)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.