Research map / Architectures and efficiency / RNNs and normalization
Towards training digitally-tied analog blocks via hybrid gradient computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces feedforward-tied energy-based models (ff-EBMs) that chain backprop through feedforward blocks and equilibrium propagation through Deep Hopfield Network energy blocks, reaching 46% top-1 on ImageNet32 with EP.
Problem
Equilibrium propagation and energy-based analog hardware are limited to fully analog, shallow models, while practical analog accelerators are hybrid with digital circuitry; end-to-end gradient computation for such mixed architectures is unclear.
Approach
Model hybrid systems as alternating feedforward (digital, trained by backprop) and energy-based (analog, trained by equilibrium propagation) blocks, and derive a BP-EP gradient chaining algorithm that passes error signals backward through both kinds of blocks, with a centered finite-beta implementation.
Key results
- EP on a 16-layer ff-EBM (DHN blocks) reaches 46.0% top-1 and 70.0% top-5 on ImageNet32, versus 41.5% top-1 for the prior best EP result cited in the paper.
- EP matches implicit differentiation (ID) on CIFAR-100 and ImageNet32 across depths, e.g. 69.3% vs 69.2% top-1 on CIFAR-100 at L=12.
- EP and ID weight gradients are near perfectly aligned (cosine similarity) across layers and blocks of a 6-block, 15-layer ff-EBM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EP with a 16-layer ff-EBM (block size 2) instead of the published EP result | ff-EBM · ImageNet32 | Top-1 accuracy (%): 36.5 -> 46.0 | Helped |
| Use EP with a 16-layer ff-EBM (block size 2) instead of the published backprop-alternative result | ff-EBM · ImageNet32 | Top-1 accuracy (%): 41.5 -> 46.0 | Helped |
| Use EP rather than ID on the 16-layer ff-EBM | ff-EBM, L=15, block size 2 · ImageNet32 | Top-1 accuracy (%): 45.5 -> 46.0 | Helped |
| Use EP rather than ID on the 16-layer ff-EBM | ff-EBM, L=15, block size 2 · CIFAR100 | Top-1 accuracy (%): 71.1$^{\pm 0.3}$ -> 71.2$^{\pm 0.2}$ | No clear effect |
| Increase depth from L=12 to L=15 with EP | ff-EBM, block size 2 · CIFAR100 | Top-1 accuracy (%): 69.3 $^{\pm 0.2}$ -> 71.2$^{\pm 0.2}$ | Helped |
| Increase depth from L=12 to L=15 with EP | ff-EBM, block size 2 · ImageNet32 | Top-1 accuracy (%): 44.7 $^{\pm 0.1}$ -> {\bfseries 46.0} $^{\pm 0.1}$ | Helped |
| Change the EB block split while keeping the total number of layers fixed (ablation) | ff-EBM trained with EP · CIFAR-10, 6 total layers | accuracy: between 89\% and 90\% | No clear effect |
| Change the EB block split while keeping the total number of layers fixed (ablation) | ff-EBM trained with EP · CIFAR-10, 12 total layers | accuracy: around $92\%$ | No clear effect |
| Use EP rather than ID | ff-EBM across tested models · CIFAR-100 and ImageNet32 | validation accuracy | No clear effect |
Limitations
- Inherits pitfalls of backprop since feedforward blocks are trained by BP.
- Experiments limited to vision tasks and relatively small/shallow architectures; not tested on full ImageNet or transformers.
- Hardware non-idealities (ADC/DAC noise, device imperfections) not modeled in experiments.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet32
Benchmarks: CIFAR-10 CIFAR-100 ImageNet32
Metrics: top-1 accuracy top-5 accuracy wall clock time cosine similarity of gradients
Baselines: implicit differentiation (recurrent backpropagation) automatic differentiation through equilibrium EP ImageNet32 prior work (36.5% top-1) EP ImageNet32 prior work (41.5% top-1)
Builds on: Equilibrium Propagation Centered EP Recurrent Backpropagation Deep Hopfield Networks
Closest papers by meaning
- Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.89)
- Training a Predictive Coding Network on ImageNet using Equilibrium Propagation (NeurIPS 2026 · similarity 0.88)
- A fast algorithm to simulate nonlinear resistive networks (ICML 2024 · similarity 0.87)
- Analog Foundation Models (NeurIPS 2025 · similarity 0.86)
- Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions (NeurIPS 2025 · similarity 0.85)
- Energy-Based Transformers are Scalable Learners and Thinkers (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.