Research map / Label-efficient and robust learning / Knowledge distillation
Learning Task-Agnostic Representations through Multi-Teacher Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a task-agnostic multi-teacher distillation objective that minimizes the conditional entropy of teacher embeddings given student embeddings, estimated with Gaussian kernels, and releases student embedders for text, molecules and vision.
Problem
Distilling several diverse pretrained embedders into one compact student that works across unseen downstream tasks without task labels; existing multi-teacher feature distillation (MSE, cosine) is task-tailored or unstable in high dimensions.
Approach
Bounds the disagreement between Bayes classifiers on student and teacher embeddings by the conditional differential entropy of teacher embeddings given the student embedding, yielding a task-agnostic loss: the negative log-likelihood of each teacher embedding under a learned per-teacher Gaussian kernel conditioned on the student embedding.
Key results
- NLL-distilled Snowflake-m (109M) average MTEB classification 76.7 vs 76.0 for MSE student and 75.5-76.0 for 3x-larger bge/GIST models.
- On TDC molecular tasks, student-2M achieves average rank 3.72, best among baselines including ChemBERTa-MTR-77M (4.79) and MSE (6.15).
- NLL-distilled students consistently outperform MSE and cosine distillation across text, molecular and vision settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MSE distillation with NLL (Gaussian kernel conditional entropy) distillation | snowflake student embedders (xs, s, m, l) from 4 teachers · MTEB classification (33-task subset) | average accuracy: +1.1 to +7.1 points across sizes | Helped |
| Distill from multiple teachers (8) instead of 1 or 2 (ablation) | GIN student on TDC ADMET and HTS · TDC regression and classification | average R2 / AUROC: regression avg R2 0.405 (1 teacher) -> 0.476 (8 teachers) | Helped |
| Use NLL instead of MSE or cosine distillation with 8 teachers | GIN student on TDC ADMET and HTS · TDC average rank | average rank: avg rank 3.72 (NLL) vs 5.13 (Cosine) and 6.15 (MSE) | Helped |
| Use NLL instead of MSE for xs-size students | snowflake xs student · MTEB classification (33-task subset) | average accuracy: 74.0 vs 72.9 | Helped |
| Distill with BBB-type task where teacher-student distribution shift is large (ablation) | GIN student with 8 teachers vs 1-2 teachers · TDC BBB classification | AUROC: slightly lower average than 1 or 2 teacher students on BBB | Hurt |
| Use NLL distillation instead of MSE on the molecular student | GIN student with 8 teachers · TDC ADMET classification and regression | average rank: consistently outperforms MSE and Cosine on all categories (text summary) | Helped |
Surprising findings
- Student-250k performs similarly to student-2M on average on TDC tasks (rank 3.94 vs 3.72), and is better on some regression tasks.
- The method does not preserve pairwise cosine structure, yet remains competitive on clustering and STS.
Limitations
- Task-specific distillation may be more effective when the target task is known.
- Precomputing teacher embeddings incurs large memory/disk cost (~100GB for the largest text teacher).
- Objective does not structure the embedding space, so clustering and STS gains are modest.
Experimental setup
Datasets: Specter T5 Amazon QA IMDB SNLI Quora QQP AG News MEDI DAIR Emotion MOSES ZINC-250k DTD
Benchmarks: MTEB (33 tasks: classification, clustering, STS) TDC ADMET and HTS (32 tasks) fine-grained vision classification
Metrics: accuracy AUROC R^2 average rank
Baselines: MSE distillation Cosine distillation CompRess bge-small-en-v1.5 GIST bge-base-en-v1.5 bge-large-en-v1.5 ChemBERTa-MTR-77M 3D-infomax GraphMVP
Builds on: Snowflake Arctic-Embed (GIST-based) SimReg CompRess Gaussian-mixture conditional entropy estimator
Closest papers by meaning
- Amalgamating Multi-Task Models with Heterogeneous Architectures (AAAI 2024 · similarity 0.89)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.87)
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.87)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
- Knowledge Composition using Task Vectors with Learned Anisotropic Scaling (NeurIPS 2024 · similarity 0.86)
- Instruction Embedding: Latent Representations of Instructions Towards Task Identification (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.