Research map / Architectures and efficiency / RNNs and normalization
Efficient Target Propagation by Deriving Analytical Solution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives closed-form analytical feedback weights (transposed-Jacobian matching) for target propagation layers so the feedback path need not be trained iteratively, matching LDRL accuracy on CIFAR-10 with about 3-5x faster feedback training.
Problem
Feedback-weight training in target propagation is computationally expensive and unstable as networks deepen, limiting scalability of biologically plausible TP compared with backpropagation.
Approach
Defines Jacobian matching losses (inverse, pseudo-inverse, transposed) and shows the transposed Jacobian matching loss (TJML) can be minimized in closed form for common layers (convolution rotated kernel, transposed FC, MaxUnpool, activations, LayerNorm, GAP, skip connections), so feedback layers skip local training iterations.
Key results
- On CIFAR-10 with Simplified-VGG, TJML-DTP reaches 85.2% test accuracy versus 89.5% for LDRL and 62.4% for FWDTP, with about 5x speed-up over LDRL.
- On LeNet, TJML-DTP matches LDRL at 99.3% MNIST test accuracy and 91.4% FMNIST, cutting training time by roughly 60-70% (3.45x and 14.29x speed-ups reported).
- On MLPMixer CIFAR-10, TJML-DTP reaches 56.9% test accuracy versus 37.3% for LDRL, with a 4x speed-up, though BP reaches 69.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | LeNet · MNIST | speedup: ×1.00 -> ×3.45 | Helped |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | LeNet · Fashion-MNIST | test accuracy: 90.7±0.26 -> 91.4±0.07 | Helped |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | LeNet · Fashion-MNIST | speedup: ×1.00 -> ×14.29 | Helped |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | LeNet · CIFAR-10 | test accuracy: 74.7±0.60 -> 76.7±0.22 | Helped |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | Simplified-VGG · CIFAR-10 | test accuracy: 89.5±0.54 -> 85.2±0.21 | Hurt |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | Simplified-VGG · CIFAR-10 | speedup: ×1.00 -> ×5.00 | Helped |
| Use analytical feedback functions satisfying TJMC instead of iterative feedback training | MLPMixer · CIFAR-10 | test accuracy: 37.3±1.48 -> 56.9±0.49; 56.1±0.35 -> 56.9±0.49 | Mixed |
| Replace the standard feedforward structure with a reversible structure (ablation) | TJML-DTP on MLPMixer · CIFAR-10 | test accuracy: 56.9±0.49 -> 42.8±0.18 | Hurt |
Surprising findings
- On Simplified-VGG the proposed method's test accuracy (85.2%) is below LDRL (89.5%) despite higher training accuracy, which the authors attribute to possible overfitting.
- The analytical feedback method is evaluated mainly on small networks; the authors note that matrix inversion cost and invertibility constraints limit IJML/PJML, so only TJML is used.
Limitations
- Analytical solutions exist only for specific layer types; matrix inversion and invertibility constraints restrict architectures.
- Test accuracy on Simplified-VGG and MLPMixer lags LDRL or BP, with possible overfitting.
- Reversible-structure extension reaches only 42.8% on MLPMixer CIFAR-10.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10
Benchmarks: MNIST Fashion-MNIST CIFAR-10
Metrics: classification accuracy relative training speed
Baselines: LDRL FWDTP Back Propagation
Builds on: Difference Target Propagation Local Difference Reconstruction Loss Fixed-Weight Difference Target Propagation Difference Reconstruction Loss
Closest papers by meaning
- Towards Scaling Difference Target Propagation by Learning Backprop Targets (ICML 2022 · similarity 0.93)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.88)
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.87)
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.87)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.85)
- Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.