Research map / Reinforcement learning / Policy gradient and actor-critic
Temporal-Difference Learning Using Distributed Error Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AD, a deep Q-learning agent whose layers each compute a local TD error (no backprop across layers), using Forward-Forward-inspired forward-in-time top-down connections and attention-like Q-value cells, evaluated on MinAtar, DMC and classic control.
Problem
How biological reward-based learning in the nucleus accumbens could perform credit assignment when dopamine TD errors are synchronously distributed and regionally homogeneous, without explicit error backpropagation.
Approach
Each layer ("AD cell") makes its own Q-value prediction and receives its own local TD error, with no error propagation between layers; upper layers send activations to lower layers forward in time, and an attention-like tanh/dot-product mechanism enables nonlinear Q regression.
Key results
- On MinAtar, AD matches DQN on Breakout and slightly surpasses DQN on Asterix and Freeway, while DQN is better on Seaquest and Space Invaders.
- On DMC tasks, AD results are close to or on par with SAC and TD-MPC2 but with marginally lower sample efficiency.
- Removing forward-in-time connections or using a single-layer cell causes devastating drops on Seaquest and Asterix.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add forward-in-time connections from upper to lower layers (ablation) | AD · MinAtar and DMC tasks | average episodic return | Helped |
| Use multiple layers instead of a single-layer cell (ablation) | AD · MinAtar and DMC tasks | average episodic return | Helped |
| Increase single-layer cell width (ablation) | AD single-layer cell · DMC tasks, Seaquest, and Asterix | average episodic return | No clear effect |
| Replace single Q-value predictions with 10 quantile predictions per cell (ablation) | AD · DMC tasks | average episodic return | Mixed |
| Use AD instead of fully connected DQN | MinAtar agents · Breakout | average episodic return | No clear effect |
| Use AD instead of fully connected DQN | MinAtar agents · Asterix and Freeway | average episodic return | Helped |
| Use AD instead of fully connected DQN | MinAtar agents · Seaquest and Space Invaders | average episodic return | Hurt |
| Use AD instead of SAC or TD-MPC2 | DMC agents · Evaluated DMC tasks | average episodic return | Mixed |
| Use AD instead of classic-control DQN | Classic control agents · Cart Pole, Mountain Car, Lunar Lander, and Acrobot | average episodic return | Helped |
| Increase depth from layer 1 to layer 4 (ablation) | AD · CIFAR-10 test | accuracy: 0.4715 -> 0.5501; 0.4750 -> 0.5588; 0.4841 -> 0.5725 | Helped |
| Increase depth from layer 1 to layer 4 (ablation) | AD · MNIST test | accuracy: 0.9848 -> 0.9867; 0.9834 -> 0.9870; 0.9834 -> 0.9874 | Mixed |
| Factorize the attention weight matrix using k=8 (ablation) | AD cells · DMC tasks | performance | No clear effect |
Surprising findings
- Forward-in-time connections from upper to lower layers are critical for performance on the most complex MinAtar tasks (Seaquest, Asterix).
- A single-layer AD cell fails on Seaquest and Asterix even when widened, suggesting multi-layer coordination matters.
Limitations
- Requires discrete action spaces; DMC tasks were discretized, and the attention weight matrix scales with action-space size, limiting scalability to large action spaces.
- Per-layer errors are an analogy; NAc neurons are not organized into layers, and asynchronous or recurrent biology is not modeled.
- Limited number of runs per task due to compute constraints.
Experimental setup
Datasets: MinAtar DeepMind Control Suite Gymnasium classic control MNIST CIFAR-10
Benchmarks: MinAtar DeepMind Control Suite Cart Pole Mountain Car Lunar Lander Acrobot MNIST CIFAR-10
Metrics: average episodic return test accuracy IQM (RLiable)
Baselines: DQN SAC TD-MPC2 ANGC CleanRL DQN
Builds on: Forward-Forward Deep Q-learning (DQN) Double Q-learning Hinton recurrent forward-forward variant
Closest papers by meaning
- A Local Temporal Difference Code for Distributional Reinforcement Learning (NeurIPS 2020 · similarity 0.88)
- Action-modulated midbrain dopamine activity arises from distributed control policies (NeurIPS 2022 · similarity 0.84)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.83)
- Real-Time Recurrent Reinforcement Learning (AAAI 2025 · similarity 0.83)
- Model Based Inference of Synaptic Plasticity Rules (NeurIPS 2024 · similarity 0.81)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.