Research map / Label-efficient and robust learning / Knowledge distillation
Pixel-Wise Contrastive Distillation
Junqiang Huang, Zichao Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pixel-level instead of image-level contrastive distillation (ablation) | PCD · COCO Mask R-CNN FPN | AP bbox: 36.5 -> 37.4 | Improved |
| keep teacher projection head via SpatialAdaptor instead of removing it (ablation) | PCD · VOC 07+12 | AP: 48.8 -> 52.1 | Improved |
| use channel-wise ReLU rather than ReLU in SpatialAdaptor (ablation) | PCD · CityScapes | mIoU: 70.1 -> 71.8 | Improved |
| add MHSA module to student (ablation) | PCD · COCO Mask R-CNN FPN | AP bbox: 37.1 -> 37.4 | Improved |
| replace MHSA with an extra prediction head of similar size (ablation) | PCD · CityScapes | mIoU: 71.8 -> 71.2 | Hurt |
| use symmetric loss on two views instead of asymmetric loss (ablation) | PCD · COCO Mask R-CNN FPN | AP bbox: 37.0 -> 37.4 | Improved |
| use SwAV, BYOL or Barlow Twins teacher instead of MoCo v3 (ablation) | PCD · COCO Mask R-CNN FPN | AP bbox: 37.4 -> 37.1 | No clear effect |
| use ViT-Base teacher (ablation) | PCD · ImageNet linear probing | top-1 accuracy: 65.1 -> 57.6 | Hurt |
| use PCD instead of supervised pre-training for ResNet-50 student | ResNet-50 · COCO Mask R-CNN FPN | AP bbox: 40.2 -> 42.4 | Improved |
Closest papers by meaning
- Masked Distillation with Receptive Tokens (ICLR 2023 · similarity 0.89)
- SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization (NeurIPS 2025 · similarity 0.87)
- Pixel Distillation: Cost-Flexible Distillation Across Image Sizes and Heterogeneous Networks (TPAMI 2024 · similarity 0.87)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.85)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.85)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
