Research map / Architectures and efficiency / RNNs and normalization
JFB: Jacobian-Free Backpropagation for Implicit Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · MNIST | average time per epoch: 28.4 -> 17.6 | Helped |
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · SVHN | average time per epoch: $\:$92.8 -> $\:$36.9 | Helped |
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · CIFAR-10 | average time per epoch: 530.9 -> {146.6} | Helped |
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · MNIST | accuracy: 99.2 -> 99.4 | Helped |
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · SVHN | accuracy: 90.1 -> 94.1 | Helped |
| Use JFB instead of Jacobian-based backpropagation | Implicit ResNet · CIFAR-10 | accuracy: 87.9 -> 93.67 | Helped |
| Train an implicit ResNet with JFB rather than its explicit ResNet counterpart | ResNet-based classifier · MNIST | accuracy: 99.4% -> 99.4% | No clear effect |
| Train an implicit ResNet with JFB rather than its explicit ResNet counterpart | ResNet-based classifier · SVHN | accuracy: 93.7% -> 94.1% | Helped |
| Train an implicit ResNet with JFB rather than its explicit ResNet counterpart | ResNet-based classifier · CIFAR-10 | accuracy: 93.0% -> 93.7% | Helped |
| Use JFB rather than Jacobian-based backpropagation | Implicit networks · MNIST, SVHN, CIFAR-10 | Jacobian matrix-vector products: $6.0 \times 10^6$ -> 0; $1.4 \times 10^{7}$ -> 0; $9.7 \times 10^8$ -> 0 | Helped |
| Use a higher-order Neumann approximation rather than JFB's zeroth-order approximation (ablation) | Implicit networks · MNIST | training time | Hurt |
| Use a higher-order Neumann approximation rather than JFB's zeroth-order approximation (ablation) | Implicit networks · MNIST | test accuracy | No clear effect |
Closest papers by meaning
- A Theoretical Framework for Inference Learning (NeurIPS 2022 · similarity 0.88)
- Bridging Discrete and Backpropagation: Straight-Through and Beyond (NeurIPS 2023 · similarity 0.87)
- Robust Implicit Networks via Non-Euclidean Contractions (NeurIPS 2021 · similarity 0.86)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.86)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.85)
- Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.