Research map / Label-efficient and robust learning / Knowledge distillation
BIRD: Behavior Induction via Representation-structure Distillation
Galen Pogoncheff, Michael Beyeler
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| match teacher representation structure (CKA) instead of activation values (Hints) (ablation) | BIRD · CIFAR-10->CIFAR-100, TinyImageNet | accuracy (%): e.g. MobileNetV2 C10->C100 51.53 -> 54.77 | Improved |
| use BIRD vs best baseline LwF | robustness transfer · C10->C100 | PGR (%): 16.2 -> 32.6 | Improved |
| linear probing or fine-tuning alone | student models · robustness transfer | accuracy (%): LP 10.95 vs none 51.31 (MobileNetV2 C10->C100) | No clear effect |
| scale student up to ResNet-152 (25x teacher) (ablation) | BIRD · weak-to-strong vision | PGR (%): 22.4% PGR for ResNet-152 | Improved |
| add BIRD loss to soft-label distillation (ablation) | GPT2-Large student · SciQ | PGR (%): 17.70 -> 24.19 | Improved |
| add BIRD loss to soft-label distillation (ablation) | GPT2-Medium student · Cosmos QA | PGR (%): 47.00 -> 42.76 | Hurt |
| add BIRD loss to soft-label distillation (ablation) | GPT2 students · BoolQ | PGR (%): 0% PGR for both | No clear effect |
| choose teacher by behavioral relevance, task relevance, complementary knowledge | teacher selection · 432 teacher-student pairs (ResNet50 student) | variance explained in PGR: R2 81.8% TinyImageNet, 85.5% CIFAR-100, 73.6% CIFAR-10 | Improved |
Closest papers by meaning
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.87)
- What Matters to You? Towards Visual Representation Alignment for Robot Learning (ICLR 2024 · similarity 0.86)
- RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers (NeurIPS 2026 · similarity 0.85)
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.85)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.85)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
