Research map / Label-efficient and robust learning / Knowledge distillation

MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities

AAAI 2025 · Label-efficient and robust learning › Knowledge distillation · approach: Patient consistent distillation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MergeNet, which transfers knowledge between heterogeneous models, tasks and modalities by low-rank re-encoding of parameter matrices and a learned low-rank parametric knowledge adapter that exchanges parameters during training.

Problem

Knowledge transfer between heterogeneous models (different architectures, tasks, and modalities) is hard because existing methods (distillation, backbone sharing) rely on shared structure or task-specific features/labels.

Approach

Treat model parameters as carriers of knowledge: low-rank decompose the transferred weight matrices and use an attention-based parameter adapter (LPKA) to map source low-rank parameters into the target model, interleaved with self-learning cycles; the adapter is removed at test time.

Key results

• Cross-structure on CIFAR-100: MergeNet reaches 66.51% top-1 for MobileNetV2 vs 63.87% vanilla and 65.48% for the best KD baseline (RKD).
• Cross-task BERT/DistilBERT on SQuAD v2.0 and IMDb: EM rises from 70.17 to 71.89 and F1 from 73.06 to 75.43; IMDb error drops from 8.02 to 7.5.
• Self knowledge transfer on MobileNetV2 (linear classifier to 16th IRB) reaches 66.48% top-1, above Tf-KD (65.43%) and USKD (65.66%).

Measured effects reported in the paper

Surprising findings

• Direct parameter sharing (putting the smaller model into the larger one) reduced MobileNetV2 accuracy, per the paper's motivating experiment.
• Transferring from the 12th layer of BERT to the 3rd layer of DistilBERT performed worse than from the 6th layer, which the authors attribute to overly advanced knowledge disrupting the target model.

Limitations

• Cross-modal experiments train on only 10% of the training set due to limited compute.
• Reported gains are modest (about 1-2 points) in most settings, and the cross-structure table compares against few recent baselines.
• Method hyperparameters such as T_cycle and rank r require tuning, and higher self-learning ratios degrade performance.

Experimental setup

Datasets: CIFAR-100 VQA v2.0 MSCOCO SQuAD v2.0 IMDb

Benchmarks: CIFAR-100 VQA v2.0 MSCOCO SQuAD v2.0 IMDb

Metrics: Top-1 accuracy Top-5 accuracy VQA accuracy R@10 EM F1 error rate parameter count

Baselines: KD RKD DKD NKD Tf-KD USKD MLP-based parameter adapter

Builds on: LoRA knowledge distillation Tf-KD USKD

Closest papers by meaning

• Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.89)
• Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.88)
• MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.87)
• Harnessing Language Model for Cross-Heterogeneity Graph Knowledge Transfer (AAAI 2025 · similarity 0.86)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model (ICLR 2024 · similarity 0.86)
• Amalgamating Multi-Task Models with Heterogeneous Architectures (AAAI 2024 · similarity 0.86)