Research map / Learning theory and algorithms / Prediction and decision losses
Sobolev Training for Neural Networks
Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, Razvan Pascanu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add target-derivative supervision to synthetic-gradient training | Synthetic-gradient classifier on CIFAR-10 · CIFAR-10 | Top 1 accuracy: 93.2% -> 93.5% | Improved |
| Add target-derivative supervision to synthetic-gradient training | ResNet50 synthetic-gradient classifier · ImageNet (one SG split) | Top 1 accuracy: 71.7% -> 72.0% | Improved |
| Add target-derivative supervision to synthetic-gradient training | ResNet50 synthetic-gradient classifier · ImageNet (one SG split) | Top 5 accuracy: 90.5% -> 90.8% | Improved |
| Add target-derivative supervision to synthetic-gradient training | ResNet50 synthetic-gradient classifier · ImageNet (three SG splits) | Top 1 accuracy: 65.7% -> 66.5% | Improved |
| Add target-derivative supervision to synthetic-gradient training | ResNet50 synthetic-gradient classifier · ImageNet (three SG splits) | Top 5 accuracy: 86.9% -> 87.4% | Improved |
| Use derivative-matching Sobolev training rather than regular regression | Two-hidden-layer ReLU regressor · Six of seven optimization benchmark functions | Test mean squared error | Improved |
| Use derivative-matching Sobolev training rather than regular regression | Two-hidden-layer ReLU regressor · Ackley | Test mean squared error | Hurt |
| Use derivative-matching Sobolev training rather than regular regression | Two-hidden-layer ReLU regressor · Ackley with enough training data | Test mean squared error | Improved |
| Add target-derivative supervision during policy distillation | Distilled Atari policy · Pong | Test policy similarity | No clear effect |
| Add target-derivative supervision during policy distillation | Distilled Atari policy · Breakout and Space Invaders | Test policy similarity | Improved |
| Use a small model resembling the main network instead of the naive direct SG model | Synthetic-gradient model · CIFAR-10 | Top 1 accuracy: 79.2% -> 88.5% | Improved |
| Add loss-value supervision to gradient-only VFBN-style training | Synthetic-gradient model · ImageNet (three SG splits) | Top 5 accuracy: 52.9% -> 87.4% | Improved |
Closest papers by meaning
- Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives (ECCV 2022 · similarity 0.87)
- Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss (ICML 2024 · similarity 0.86)
- Approximation with CNNs in Sobolev Space: with Applications to Classification (NeurIPS 2022 · similarity 0.86)
- Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives (NeurIPS 2023 · similarity 0.85)
- Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods (JMLR 2025 · similarity 0.84)
- Why Deep Neural Networks for Function Approximation? (ICLR 2017 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
