Research map / Architectures and efficiency / RNNs and normalization
K-prop: Deep Online Learning by Backpropagating Temporal Kernels
Bingkun Liu, Paul Haider, Federico Benitez, Mihai A. Petrovici, Walter Senn
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes K-prop, an online learning method for leaky (self-recurrent) neural networks that backpropagates through temporal kernels to handle complex temporal gradient dependencies.
Problem
Efficient and effective online training of self-recurrent (leaky integration) neural networks is lacking because the temporal dependencies of the gradient are complex.
Approach
Backpropagates gradients through temporal kernels of leaky neurons to enable online learning in leaky neural networks, aiming to avoid the complex temporal gradient dependencies that hinder existing online methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use K-prop instead of OSTL for online training | leaky network · Google Speech Commands (20 commands) | test accuracy (%): 90.61 -> 91.52 | Improved |
| Use K-prop instead of OSTL for online training | leaky network Lk4 · MNIST1D (Lk4) | test accuracy (%): 96.08 -> 96.53 | Improved |
| Use K-prop instead of OSTL for online training | leaky network Lk5 · MNIST1D (Lk5) | test accuracy (%): 97.03 -> 97.56 | Improved |
| Replace exact BPTT with K-prop | leaky network · Google Speech Commands (20 commands) | test accuracy (%): 92.47 -> 91.52 | Hurt |
| Freeze progressively more early layers during training (ablation) | K-prop Lk5 · MNIST1D (Lk5) | accuracy | Hurt |
| Use low-pass-filtered rho' as surrogate instead of instantaneous rho' (ablation) | K-prop · gradient cosine dissimilarity, random networks on MNIST1D | cosine dissimilarity to BPTT | Improved |
| Use sigmoid instead of ReLU or tanh activations (ablation) | K-prop · gradient cosine dissimilarity, random networks on MNIST1D | cosine dissimilarity to BPTT | Improved |
| Increase network depth (distance from output layer) (ablation) | K-prop gradient · gradient cosine dissimilarity, random networks on MNIST1D | cosine dissimilarity to BPTT | Hurt |
| Increase neuron time constant tau (ablation) | K-prop gradient · gradient cosine dissimilarity, random networks on MNIST1D | cosine dissimilarity to BPTT | Mixed |
| Apply weight updates at discrete intervals instead of every step (ablation) | online learning · N-sum / sequence tasks | learning stability | Improved |
| Add Gaussian noise to eligibility traces Z (ablation) | K-prop Lk4 · MNIST1D | accuracy (%): 96.53 -> 94.5 (sigma=1), 92.3 (sigma=10) | No clear effect |
Closest papers by meaning
- Approximating Real-Time Recurrent Learning with Random Kronecker Factors (NeurIPS 2018 · similarity 0.86)
- Online learning of long-range dependencies (NeurIPS 2023 · similarity 0.85)
- Training Recurrent Neural Networks Online by Learning Explicit State Variables (ICLR 2020 · similarity 0.84)
- Exploring the Promise and Limits of Real-Time Recurrent Learning (ICLR 2024 · similarity 0.84)
- Kernel Representation Learning for Time Sequence: Algorithm, Theory, and Applications (AAAI 2025 · similarity 0.83)
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
