Research map / Label-efficient and robust learning / Knowledge distillation

Scaling Multi-Teacher Distillation for Digital Pathology

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows vanilla multi-teacher distillation scales poorly with more teachers for pathology encoders, and proposes teacher-specific tokens with structured attention, MoE tail, and InfoNCE to yield OMNI student encoders (7M-330M) that match or beat 1.1B teachers.

Problem

Vanilla multi-teacher distillation (MTD) does not improve, and even degrades, downstream performance as the number of teachers grows, since a single shared CLS token must reconstruct many distinct teacher embedding spaces.

Approach

Replace the single shared CLS token with one learnable teacher-specific token per teacher, use a block-structured attention mask that isolates teacher tokens from each other at aggregation, add MoE tail blocks and a per-teacher InfoNCE loss, and mean-pool teacher tokens at inference.

Key results

• OMNI-L reaches 87.8% tile-level F1 and 63.5% slide-level Bacc, the best of all models, outperforming teachers such as UNI2-h (681M) and H-optimus-1 (1.1B).
• Vanilla MTD cosine reconstruction drops from 0.90 at 1 teacher to 0.83 at 10 teachers, while the proposed method retains 0.87 at 10 teachers.
• OMNI-B reaches 87.1% tile F1 with 21.8 GFLOPs, versus 180.4 GFLOPs for UNI2-h, while removing the InfoNCE, MoE, or attention mask each reduces the average score by 1.3 points.

Surprising findings

• Gradient conflict analysis finds no major conflicts between teachers (mean cosine similarity 0.162), and training losses are stable, so the scaling failure is attributed to the single-CLS bottleneck rather than optimization or conflicts.
• Full isolation of teacher tokens (blocking all cross-teacher attention paths) slightly hurts performance (87.1 to 86.7 F1), so partial isolation suffices.

Limitations

• Mean pooling is the default aggregation; learned teacher weighting was only explored on BRACS and is not studied in depth.
• Only the global CLS-level and standard patch losses are used; teacher-specific dense representations were not explored.
• Clinical use would require further validation beyond the 39 benchmark tasks.

Experimental setup

Datasets: TCGA (10M tiles, 10K WSIs) THUNDER PathoBench CPTAC BRACS IMP

Benchmarks: THUNDER PathoBench CPTAC IMP colorectal (PathoBench)

Metrics: F1 balanced accuracy mean rank F1 drop under PGD attack ECE cosine reconstruction similarity

Baselines: UNI2-h Virchow2 H-optimus-1 KEEP Prov-GigaPath CONCH 1.5 Hibou-B Kaiko-ViT-B/8 GPFM H0-mini Vanilla MTD UNIC

Builds on: vanilla multi-teacher distillation DINOv2 Switch Transformer load-balancing loss prompt tuning UNIC DUNE

Closest papers by meaning

• Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)
• Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.87)
• Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.87)
• Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.87)
• How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · similarity 0.86)
• CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.86)