AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Sobolev Training for Neural Networks

Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, Razvan Pascanu

NeurIPS 2017 · Learning theory and algorithms › Prediction and decision losses · approach: Theory of neural network sensitivity · 111 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add target-derivative supervision to synthetic-gradient trainingSynthetic-gradient classifier on CIFAR-10 · CIFAR-10Top 1 accuracy: 93.2% -> 93.5%Improved
Add target-derivative supervision to synthetic-gradient trainingResNet50 synthetic-gradient classifier · ImageNet (one SG split)Top 1 accuracy: 71.7% -> 72.0%Improved
Add target-derivative supervision to synthetic-gradient trainingResNet50 synthetic-gradient classifier · ImageNet (one SG split)Top 5 accuracy: 90.5% -> 90.8%Improved
Add target-derivative supervision to synthetic-gradient trainingResNet50 synthetic-gradient classifier · ImageNet (three SG splits)Top 1 accuracy: 65.7% -> 66.5%Improved
Add target-derivative supervision to synthetic-gradient trainingResNet50 synthetic-gradient classifier · ImageNet (three SG splits)Top 5 accuracy: 86.9% -> 87.4%Improved
Use derivative-matching Sobolev training rather than regular regressionTwo-hidden-layer ReLU regressor · Six of seven optimization benchmark functionsTest mean squared errorImproved
Use derivative-matching Sobolev training rather than regular regressionTwo-hidden-layer ReLU regressor · AckleyTest mean squared errorHurt
Use derivative-matching Sobolev training rather than regular regressionTwo-hidden-layer ReLU regressor · Ackley with enough training dataTest mean squared errorImproved
Add target-derivative supervision during policy distillationDistilled Atari policy · PongTest policy similarityNo clear effect
Add target-derivative supervision during policy distillationDistilled Atari policy · Breakout and Space InvadersTest policy similarityImproved
Use a small model resembling the main network instead of the naive direct SG modelSynthetic-gradient model · CIFAR-10Top 1 accuracy: 79.2% -> 88.5%Improved
Add loss-value supervision to gradient-only VFBN-style trainingSynthetic-gradient model · ImageNet (three SG splits)Top 5 accuracy: 52.9% -> 87.4%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy