Research map / Label-efficient and robust learning / Knowledge distillation
Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers
Anh Nguyen, Ngan Nguyen, Hong Duc Vu, Trung Dao, Viet Nguyen, Minh Quan Dao, Kien Nguyen, Tran Bao Chi Tran, Phong Nguyen, Khoi Nguyen, Cuong Pham, Dimitris N. Metaxas and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from SD 3.5 Medium teacher through Bridge into one-step SD 1.5 student | SD 1.5 DMD2 student · HPSv3 | HPSv3: 5.37 -> 9.42 | Improved |
| distill into SD 1.5 via Bridge | SD 1.5 DMD2 student · ImageReward | ImageReward: -0.29 -> 0.62 | Improved |
| average parameters of five teacher-specific distilled students (ablation) | SD 1.5 Bridge students · HPSv3 | HPSv3: 5.37 -> 10.53 | Improved |
| average parameters of five distilled students (ablation) | SD 2.1 Bridge students · HPSv3 | HPSv3: 6.42 -> 9.75 | Improved |
| replace pruned SD 3.5 initialization (30% sparsity) with Bridge distillation to SD 1.5 (ablation) | one-step student · HPSv3 | HPSv3: 2.76 -> 9.42 | Improved |
| initialize student from multi-step SD 1.5 instead of DMD2 one-step checkpoint (ablation) | SD 1.5 student · HPSv3 / HPSv2 | HPSv3: HPSv3 9.42 -> 9.22; HPSv2 28.30 -> 28.44 | No clear effect |
| add Attention Fidelity loss to latent reconstruction for Bridge training (ablation) | Bridge · decoded-image fidelity (L1/PSNR/SSIM) | reconstruction fidelity | Improved |
| add E-LatentLPIPS to L1 loss (ablation) | Bridge · decoded-image fidelity | reconstruction fidelity | No clear effect |
| add frozen Student-decoder spatial prior to SwinIR projector (ablation) | Bridge · decoded-image fidelity | reconstruction fidelity | Improved |
| use SwinIR/UNet backbone instead of MLP for the projector (ablation) | Bridge · decoded-image fidelity | reconstruction fidelity | Improved |
| jointly train Bridge with the student during distillation instead of freezing (ablation) | Bridge distillation · training stability | stability | Hurt |
Closest papers by meaning
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations (NeurIPS 2025 · similarity 0.86)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.86)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.86)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.86)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
