Research map / Optimization / Optimizers
Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a theory of analog (AIMC) gradient training under generic asymmetric, non-linear resistive-element response functions, showing Analog SGD has an implicit penalty and proposing Residual Learning (a bilevel method generalizing Tiki-Taka) that converges exactly under zero-shifted symmetric points, plus v2 variants for noisy IO and limited granularity.
Problem
Analog in-memory computing training with pulse updates is governed by asymmetric, non-linear response functions, which bias gradient descent and cause inexact convergence; existing theory covers only special linear responses.
Approach
Model analog update as a compact dynamics with symmetric and asymmetric components F and G; show Analog SGD implicitly minimizes a penalized objective attracting weights to symmetric points; propose Residual Learning, a bilevel scheme with a residual array P whose stationary point is 0 and that is transferred into the main array, aligning the algorithmic stationary point with the physical symmetri
Key results
- On CIFAR100 ResNet18 with power response, Residual Learning v2 reaches 79.83% vs 68.98% for Analog SGD and 81.12% for digital SGD.
- On CIFAR10 ResNet18, Analog SGD reaches 84.47% while Tiki-Taka/Residual Learning reaches 94.81% vs 95.43% digital SGD.
- Residual Learning keeps accuracy drop under 1% on MNIST FCN and CNN, while Analog SGD converges more slowly and plateaus lower for CNN (about 80%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Residual Learning / Tiki-Taka instead of Analog SGD | ResNet training with power response · CIFAR100, ResNet18 | test accuracy: 68.98±1.01 -> 76.17±0.23 | Helped |
| Add a digital buffer and threshold-based transfer (Residual Learning v2) | Residual Learning · CIFAR100 | test accuracy | Helped |
| Use Residual Learning v2 instead of Tiki-Taka v2 | ResNet training with power response · CIFAR100 | test accuracy | Helped |
| Use Residual Learning v2 instead of Tiki-Taka v2 | ResNet training with power response · CIFAR10 | test accuracy | Mixed |
| Increase the mixing coefficient γ from 0 to 0.1 (ablation) | Residual Learning · CIFAR10/CIFAR100 ResNet training | test accuracy | Helped |
| Increase the mixing coefficient γ from 0.1 to 0.4 (ablation) | Residual Learning · CIFAR10/CIFAR100 ResNet training | test accuracy | No clear effect |
| Use a non-zero mixing coefficient γ (ablation) | Tiki-Taka / Residual Learning · analog training | convergence rate: O(1/(1 - 33P_max^2/τ^2)) -> O(1/(1 - P_max^2/τ^2)) | Helped |
| Use Tiki-Taka instead of Analog SGD | ResNet training with power response · CIFAR10/CIFAR100 | test accuracy | Helped |
| Use Residual Learning instead of Analog SGD | FCN/CNN · MNIST | test accuracy: <1% drop | Helped |
| Perturb the linear response parameter c_Lin from 0 to 0.1–0.3 (ablation) | Tiki-Taka · synthetic response-function simulation | training accuracy: 0.1 to 0.3 | Hurt |
Surprising findings
- Tiki-Taka can become worse than Analog SGD when a small linear asymmetry term c_Lin is perturbed from 0.1 to 0.3, showing fragility of existing convergence guarantees.
Limitations
- Analysis considers only asymmetric response, limited granularity, and noisy IO; other hardware imperfections are not covered.
- Convergence theory assumes strong convexity, which is needed for uniqueness of the optimal point and may not hold for neural networks.
- Residual Learning doubles analog memory requirement.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100
Benchmarks: MNIST CIFAR10 CIFAR100
Metrics: test accuracy
Baselines: Analog SGD Tiki-Taka Tiki-Taka v2 Digital SGD
Builds on: Tiki-Taka zero-shifting analog SGD
Closest papers by meaning
- Analog Foundation Models (NeurIPS 2025 · similarity 0.90)
- Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.89)
- Towards training digitally-tied analog blocks via hybrid gradient computation (NeurIPS 2024 · similarity 0.85)
- A fast algorithm to simulate nonlinear resistive networks (ICML 2024 · similarity 0.85)
- Rethinking Neural Nonlinearity as Gating (NeurIPS 2026 · similarity 0.84)
- Reshape and Adapt for Output Quantization (RAOQ): Quantization-aware Training for In-memory Computing Systems (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.