AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Gradients as Features for Deep Representation Learning

Fangzhou Mu, Yingyu Liang, Yin Li

ICLR 2020 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 16 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add pre-trained gradient features to activation featuresFull model with BiGAN encoder · SVHN, CIFAR-10, CIFAR-100accuracy: 82.04 -> 89.96; 62.87 -> 70.51; 34.30 -> 39.37Improved
combine activation features with gradient featuresFull model with VAE encoder · SVHN, CIFAR-10, CIFAR-100accuracy: 81.95 -> 90.90; 52.05 -> 61.16; 29.20 -> 34.83Mixed
combine activation and gradient featuresFull model with ResNet50 Jigsaw · VOC07 and COCO2014mAP: 57.83 -> 61.70; 42.10 -> 45.62Improved
combine activation and gradient featuresFull model with ResNet18 ImageNet · VOC07 and COCO2014mAP: 83.59 -> 84.95; 62.98 -> 66.45Improved
use full model rather than fine-tuningResNet18 ImageNet transfer · VOC07 and COCO2014mAP: 84.14 -> 84.95; 63.43 -> 66.45Improved
use full model rather than fine-tuningResNet50 Jigsaw self-supervision · VOC07 and COCO2014mAP: 67.88 -> 61.70; 52.66 -> 45.62Hurt
use pre-trained rather than random gradient parameters (ablation)Full model · CIFAR-10, BiGAN encoderaccuracy: 62.83 -> 70.51Improved
add gradients derived from pre-trained parameters rather than random parameters (ablation)Full model · VOC07, ImageNet-pre-trained ResNet18mAP: 82.84 -> 83.50No clear effect
use pre-trained rather than random bottom-layer parameters for gradient features (ablation)Full model · CIFAR-10, BiGAN encoderaccuracy: 62.83 -> 70.51Improved
increase gradient-feature layers from conv5 to conv4-5 or conv3-5 (ablation)Full model · CIFAR-10, BiGAN encoderaccuracy: 70.51; 70.78; 71.37No clear effect
increase gradient-feature layers from layer4 block2 to layer4 block1-2 (ablation)Full model · VOC07, ImageNet-pre-trained ResNet18mAP: 83.50 -> 83.40Mixed
use gradient features alone instead of activation features (ablation)Gradient baseline · CIFAR-10, BiGAN encoderaccuracy: 62.87 -> 70.14Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy