Research map / Label-efficient and robust learning / Knowledge distillation
Complementary Relation Contrastive Distillation
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill anchor-based relations with CRCD | student networks · CIFAR100, seven teacher-student pairs | top-1 accuracy | Helped |
| distill complementary feature and gradient relations together instead of either alone (ablation) | CRCD · CIFAR100, seven teacher-student pairs | top-1 accuracy: 76.18 -> 76.67; 76.27 -> 76.67; 75.26 -> 75.95; 75.24 -> 75.95; 72.70 -> 73.21; 72.64 -> 73.21; 72.02 -> 72.33; 71.73 -> | Helped |
| replace low-dimensional similarity relations with learned high-dimensional cross-space relations (ablation) | relation distillation · CIFAR100, three teacher-student pairs | top-1 accuracy: 71.93 -> 72.70; 71.54 -> 72.02; 74.15 -> 74.69 | Helped |
| use relation contrastive loss rather than L2 loss for feature relations (ablation) | CRCD relation modeling · CIFAR100, three teacher-student pairs | top-1 accuracy: 71.93 -> 72.70; 71.54 -> 72.02; 74.15 -> 74.69 | Helped |
| use learnable linear or nonlinear projections in the critic instead of identity mapping (ablation) | CRCD critic · CIFAR100, three teacher-student pairs | top-1 accuracy: 72.35 -> 72.52; 71.84 -> 71.98; 74.25 -> 74.49 | Helped |
| use CRCD loss instead of triplet-margin or contrastive logistic loss (ablation) | feature-relation distillation · CIFAR100, three teacher-student pairs | top-1 accuracy: 72.21 -> 72.70; 72.15 -> 72.70; 74.25 -> 74.69; 71.83 -> 72.02; 71.72 -> 72.02; 74.07 -> 74.69 | Helped |
| use L_NCE instead of CRCD loss (ablation) | feature-relation distillation · CIFAR100 (ResNet110→ResNet20) | top-1 accuracy: 72.02 -> 72.09 | Helped |
| use queue sampling rather than random negative sampling (ablation) | CRCD · CIFAR100 (ResNet56→ResNet20) | top-1 accuracy | Helped |
| increase the number of negative samples (ablation) | CRCD · CIFAR100 (ResNet56→ResNet20) | top-1 accuracy | Helped |
| use extremely high or low temperature (ablation) | CRCD · CIFAR100 (ResNet56→ResNet20) | top-1 accuracy | Hurt |
| reduce critic projection dimension to 64-d (ablation) | CRCD critic · CIFAR100, three teacher-student pairs | top-1 accuracy: 72.45 vs 72.70; 71.92 vs 72.02; 74.34 vs 74.69 | Hurt |
| distill CRCD on ImageNet | ResNet34→ResNet18 · ImageNet validation | top-1 error rate: 28.38 -> 28.04 | Helped |
Closest papers by meaning
- Contrastive Representation Distillation (ICLR 2020 · similarity 0.91)
- Relational Knowledge Distillation (CVPR 2019 · similarity 0.90)
- Wasserstein Contrastive Representation Distillation (CVPR 2021 · similarity 0.90)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.87)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.