Research map / Architectures and efficiency / RNNs and normalization
Gradients as Features for Deep Representation Learning
Fangzhou Mu, Yingyu Liang, Yin Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pre-trained gradient features to activation features | Full model with BiGAN encoder · SVHN, CIFAR-10, CIFAR-100 | accuracy: 82.04 -> 89.96; 62.87 -> 70.51; 34.30 -> 39.37 | Improved |
| combine activation features with gradient features | Full model with VAE encoder · SVHN, CIFAR-10, CIFAR-100 | accuracy: 81.95 -> 90.90; 52.05 -> 61.16; 29.20 -> 34.83 | Mixed |
| combine activation and gradient features | Full model with ResNet50 Jigsaw · VOC07 and COCO2014 | mAP: 57.83 -> 61.70; 42.10 -> 45.62 | Improved |
| combine activation and gradient features | Full model with ResNet18 ImageNet · VOC07 and COCO2014 | mAP: 83.59 -> 84.95; 62.98 -> 66.45 | Improved |
| use full model rather than fine-tuning | ResNet18 ImageNet transfer · VOC07 and COCO2014 | mAP: 84.14 -> 84.95; 63.43 -> 66.45 | Improved |
| use full model rather than fine-tuning | ResNet50 Jigsaw self-supervision · VOC07 and COCO2014 | mAP: 67.88 -> 61.70; 52.66 -> 45.62 | Hurt |
| use pre-trained rather than random gradient parameters (ablation) | Full model · CIFAR-10, BiGAN encoder | accuracy: 62.83 -> 70.51 | Improved |
| add gradients derived from pre-trained parameters rather than random parameters (ablation) | Full model · VOC07, ImageNet-pre-trained ResNet18 | mAP: 82.84 -> 83.50 | No clear effect |
| use pre-trained rather than random bottom-layer parameters for gradient features (ablation) | Full model · CIFAR-10, BiGAN encoder | accuracy: 62.83 -> 70.51 | Improved |
| increase gradient-feature layers from conv5 to conv4-5 or conv3-5 (ablation) | Full model · CIFAR-10, BiGAN encoder | accuracy: 70.51; 70.78; 71.37 | No clear effect |
| increase gradient-feature layers from layer4 block2 to layer4 block1-2 (ablation) | Full model · VOC07, ImageNet-pre-trained ResNet18 | mAP: 83.50 -> 83.40 | Mixed |
| use gradient features alone instead of activation features (ablation) | Gradient baseline · CIFAR-10, BiGAN encoder | accuracy: 62.87 -> 70.14 | Improved |
Closest papers by meaning
- Characterizing emergent representations in a space of candidate learning rules for deep networks (NeurIPS 2020 · similarity 0.88)
- Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation (NeurIPS 2018 · similarity 0.87)
- The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks (NeurIPS 2026 · similarity 0.86)
- When Representations Align: Universality in Representation Learning Dynamics (ICML 2024 · similarity 0.86)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
