Research map / Architectures and efficiency / Attention and transformers
Co-Advise: Cross Inductive Bias Distillation
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from convolution and involution teachers with separate tokens instead of DeiT-style distillation | CivT-Ti on ImageNet-1k · ImageNet-1k | Top-1 accuracy (%): 74.5 -> 74.9 | Helped |
| Distill from convolution and involution teachers with separate tokens instead of DeiT-style distillation | CivT-S on ImageNet-1k · ImageNet-1k | Top-1 accuracy (%): 81.2 -> 82.0 | Helped |
| Add a RedNet-26 teacher to ResNet-18 distillation (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 86.5 -> 88.0 | Helped |
| Add another convolution teacher of the same architectural type (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 86.5 -> 87.2 | Helped |
| Replace a second convolution teacher with an involution teacher (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 87.2 -> 88.0 | Helped |
| Use three tokens rather than one while distilling from both teacher types (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 83.5 -> 88.0; 4.5% | Helped |
| Use one token and distill from only ResNet-18 (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 81.8 -> 81.9 | No clear effect |
| Use one token and distill from only RedNet-26 (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 81.8 -> 80.7 | Hurt |
| Add ResNet-18 and RedNet-26 teachers (ablation) | ResNet-10 · ImageNet-100 | Top-1 accuracy (%): 81.5 -> 83.4 | Helped |
| Add ResNet-18 and RedNet-26 teachers (ablation) | Mixer-Ti · ImageNet-100 | Top-1 accuracy (%): 80.5 -> 82.3 | Helped |
| Distill from both teacher types rather than one teacher (ablation) | Transformer-Ti · ImageNet-100 | Top-1 accuracy (%): 86.5 -> 88.0 | Helped |
Closest papers by meaning
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.90)
- No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.88)
- Training the Untrainable: Introducing Inductive Bias via Representational Alignment (NeurIPS 2025 · similarity 0.88)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.87)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.