Research map / Label-efficient and robust learning / Knowledge distillation

You Don’t Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RPKD, a knowledge distillation method that projects teacher and student features onto a shared set of randomly initialized, frozen prototype vectors instead of aligning their representation spaces.

Problem

Knowledge distillation typically assumes teacher and student representations must be compatible or explicitly aligned, which often requires architecture-specific adapters or class-specific alignment.

Approach

Discards the aligned-representation assumption by projecting both teacher and student networks into a shared set of randomly initialized, frozen prototype vectors, requiring no architecture-specific adapters or class-specific alignment.

Surprising findings

• Compatible or explicitly aligned teacher and student feature spaces are not required for effective distillation, per the abstract's premise.

Closest papers by meaning

• FreeKD: Knowledge Distillation via Semantic Frequency Prompt (CVPR 2024 · similarity 0.89)
• Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.88)
• Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation (AAAI 2025 · similarity 0.87)
• Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.87)
• ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
• Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.87)