Research map / Optimization / Optimizers
Towards Exact Gradient-based Training on Analog In-memory Computing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a discrete-time dynamic model for SGD on asymmetric linear analog devices, proves analog SGD converges only inexactly (with matching lower bound), and shows the Tiki-Taka two-array algorithm converges exactly to a critical point.
Problem
Gradient-based training on analog in-memory computing (AIMC) devices with asymmetric pulse updates: standard SGD theory fails, and analog SGD converges to a biased point with non-vanishing asymptotic error.
Approach
Model analog SGD on asymmetric linear devices as SGD plus a weight-dependent bias term (|grad|*W/tau), prove an asymptotic error that is intrinsic via a lower bound, and show that the two-array Tiki-Taka heuristic reduces the drift from O(alpha) to O(alpha*beta), eliminating the asymptotic error.
Key results
- Analog SGD exhibits an asymptotic error that does not shrink as the learning rate decreases, contrary to digital SGD theory, and the error decreases as tau increases.
- A matching lower bound shows the analog SGD error term 4*sigma^2*S_K is intrinsic, with Omega(4*sigma^2*S_K + 1/sqrt(K)) rate.
- On MNIST, Tiki-Taka drops accuracy by less than 1% relative to digital SGD on both FCN and CNN, while Analog SGD drops by more than 6% on CNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tiki-Taka instead of Analog SGD | Analog training · Non-convex stochastic optimization | asymptotic error: $O(\sigma^2 S_K) -> 0$ | Helped |
| Use Tiki-Taka instead of Analog SGD | Analog training · Non-convex stochastic optimization | convergence rate: $O\left(\sqrt{\frac{\sigma^2}{K}}\frac{1}{1-W_{\max}^2/\tau^2}\right) -> O\left(\sqrt{\frac{\sigma^2}{K}}\frac{1}{1-33P_ | Mixed |
| Increase device parameter $\tau$ (ablation) | Analog SGD · Least-squares task | asymptotic error | Helped |
| Increase noise variance $\sigma^2$ (ablation) | Analog SGD · Least-squares task | asymptotic error | Hurt |
| Change the initialization (ablation) | Analog SGD · Least-squares task | asymptotic error | No clear effect |
| Use Tiki-Taka instead of Analog SGD | MNIST FCN · MNIST | test accuracy: drops by $<1\%$ | Helped |
| Use Tiki-Taka instead of Analog SGD | MNIST CNN · MNIST | test accuracy: drops by $> 6\%$ | Helped |
| Reduce the variance using Tiki-Taka | Analog training · MNIST FCN and CNN | test accuracy: <1\% | Helped |
| Use analog training instead of Digital SGD | Analog SGD and Tiki-Taka · CIFAR10 Resnet | test accuracy | Helped |
Surprising findings
- On CIFAR10 ResNet, both Analog SGD and Tiki-Taka outperform Digital SGD, which the authors conjecture is due to analog noise improving robustness to outlier data.
Limitations
- Analysis is restricted to the asymmetric linear device (ALD) model.
- Assumptions such as bounded saturation (W_max < tau) must hold or be enforced by clipping.
- Tiki-Taka is heuristic and the theory relies on noise being non-zero in absolute mean.
Experimental setup
Datasets: MNIST CIFAR10 synthetic least-squares problem
Benchmarks: MNIST CIFAR10
Metrics: test accuracy squared gradient norm asymptotic error
Baselines: Analog SGD Digital SGD Tiki-Taka (TT-v1)
Builds on: Tiki-Taka stochastic gradient descent AIHWKIT
Closest papers by meaning
- Towards training digitally-tied analog blocks via hybrid gradient computation (NeurIPS 2024 · similarity 0.89)
- Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions (NeurIPS 2025 · similarity 0.89)
- Analog Foundation Models (NeurIPS 2025 · similarity 0.88)
- Energy-based learning algorithms for analog computing: a comparative study (NeurIPS 2023 · similarity 0.88)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
- Rethinking Neural Nonlinearity as Gating (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.