Research map / Label-efficient and robust learning / Recognition and long-tail
Why Do Better Loss Functions Lead to Less Transferable Features?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax cross-entropy with alternative objectives | ResNet-50 ImageNet pretraining · ImageNet ILSVRC 2012 val | Top-1 accuracy: 77.0 -> 77.9 | Helped |
| Replace softmax cross-entropy with alternative objectives | ResNet-50 fixed-feature transfer · Food-101 | Linear transfer accuracy: 74.6 -> 62.0 | Hurt |
| Pretrain with alternative objectives rather than softmax | ResNet-50 fine-tuning · Transfer datasets average | Accuracy: 86.2 -> 86.4 | No clear effect |
| Lower cosine-softmax temperature from 0.05 to 0.01 (ablation) | Cosine softmax ResNet-50 · ImageNet ILSVRC 2012 val | Top-1 accuracy: 77.6 -> 74.9 | Hurt |
| Lower cosine-softmax temperature from 0.05 to 0.01 (ablation) | Cosine softmax ResNet-50 · Food-101 linear transfer | Accuracy: 62.8 -> 73.4 | Helped |
| Raise cosine-softmax temperature from 0.01 to 0.08 (ablation) | Cosine softmax ResNet-50 · Food-101 linear transfer | Accuracy: 73.4 -> 53.7 | Hurt |
| Combine alternative losses and final-layer regularizers (ablation) | ResNet-50 ImageNet training · ImageNet holdout set | Top-1 accuracy: 80.6; 80.6 | No clear effect |
| Add AutoAugment to training (ablation) | Softmax ResNet-50 · ImageNet | Top-1 accuracy: 77.0 ± 0.06 -> 77.7 ± 0.05 | Helped |
| Increase class separation through alternative objectives | ImageNet-pretrained fixed features · Natural image transfer tasks | Linear transfer accuracy: Spearman's ρ = -0.93 | Hurt |
| Use alternative objectives instead of softmax cross-entropy | ResNet-50 ImageNet pretraining · ImageNet-V2 | Accuracy: 65.0 ± 0.1 -> 65.9 ± 0.1 | Helped |
Closest papers by meaning
- How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model (NeurIPS 2025 · similarity 0.90)
- Do Better ImageNet Models Transfer Better? (CVPR 2019 · similarity 0.89)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.86)
- Investigating the Benefits of Projection Head for Representation Learning (ICLR 2024 · similarity 0.86)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.85)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.