Research map / Optimization / Optimizers

BP-Modified Local Loss for Efficient Training of Deep Neural Networks

ICLR 2025 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a BP-modified local loss method that uses true backpropagation gradients to correct local-loss gradients, aiming to speed up convergence and reduce memory during training of deep networks.

Problem

Local loss training methods (GIM, LoCo, Forward-Forward) reduce memory for training large models but often converge slowly or fail to converge.

Approach

Uses the true backpropagation gradient to modify the local loss gradient, improving convergence of local-loss training while keeping the memory benefits of local learning.

Experimental setup

Baselines: GIM LoCo Forward-Forward

Builds on: GIM LoCo Forward-Forward

Closest papers by meaning

• Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · similarity 0.87)
• Scaling Forward Gradient With Local Losses (ICLR 2023 · similarity 0.87)
• On the Local Hessian in Back-propagation (NeurIPS 2018 · similarity 0.87)
• Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.86)
• Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
• BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning (NeurIPS 2024 · similarity 0.83)