Research map / Optimization / Implicit bias and generalization
Curl Descent : Non-Gradient Learning Dynamics with Sign-Diverse Plasticity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that sign-diverse (rule-flipped) plasticity yields non-gradient "curl" learning dynamics in linear and nonlinear student-teacher networks, analyzing when the solution manifold stays stable and finding that a few flipped readout weights can speed convergence.
Problem
Whether biological learning rules with sign-diverse plasticity (Hebbian/anti-Hebbian, Dale's law E-I networks) can still optimize a loss, given that their dynamics may not be gradient flows on any objective.
Approach
Introduce "curl descent," which flips the sign of the weight update for a random subset of presynaptic neurons in a two-layer student-teacher network, creating non-gradient rotational dynamics; use Jacobian eigenvalue analysis with random matrix theory to find stability boundaries as a function of compression ratio and flipped fraction, and show that curl terms can speed convergence in nonlinear n
Key results
- Expansive networks (compression ratio c<1) remain stable to hidden-layer curl terms; instability appears above c≈0.3 for the hidden layer, while readout curl stability holds for at most half the readout neurons flipped.
- In nonlinear tanh student-teacher networks, flipping a single readout weight (of 1010 parameters) led to faster convergence than gradient descent (p<.01 for N_train≥500).
- Hidden-layer curl terms in simulations produce chaotic weight dynamics once the solution manifold loses stability, while readout-layer curl terms can still reach low test error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rule-flipped plasticity to the hidden layer | Curl descent in linear student-teacher networks · Solution manifold stability across compression ratios | stability: c \approx 0.3 | Hurt |
| Add rule-flipped plasticity to the readout layer | Curl descent in linear student-teacher networks · Solution manifold stability | stability: at most half | Hurt |
| Add rule-flipped plasticity to the hidden layer beyond the stability boundary | Curl descent in linear networks · Unstable regime | learning dynamics | Hurt |
| Add rule-flipped plasticity to the readout layer beyond the stability boundary | Curl descent in linear networks · Unstable regime | test error | Mixed |
| Flip the learning-rule sign for a single readout weight | Curl descent in tanh networks · Nonlinear student-teacher learning | convergence speed | Helped |
| Use curl descent instead of gradient descent in the high-data regime | Tanh networks with one rule-flipped readout weight · N_train ≥ 500 | convergence speed: p<.01 | Helped |
| Use curl descent instead of gradient descent below the critical teacher-weight initialization scale | Tanh networks · Teacher complexity below the critical initialization scale | convergence speed: p<.01 | Helped |
| Increase the fraction of rule-flipped readout neurons (ablation) | Curl descent in tanh networks · Convergence speed as flipped-neuron fraction varies | convergence speed | Hurt |
| Use curl descent with up to half of readout weights rule-flipped | Tanh networks · Readout rule-flipped fraction ≤ 50% | convergence speed: ≤ 50%; p<.01 | Helped |
| Flip the hidden neuron's plasticity rule in the two-neuron network | Curl flow in the two-neuron network · Solution manifold | stability: half | Hurt |
| Use curl flow rather than gradient flow near the saddle's stable manifold | Two-neuron network · Convergence from saddle-associated initializations | convergence speed | Helped |
| Initialize weights at small amplitude under curl flow | Two-neuron network · Small-amplitude initializations | convergence | Hurt |
Surprising findings
- Curl descent can converge faster than gradient descent in nonlinear networks by escaping saddles through temporary loss ascent.
- Readout-layer curl terms yield low test error even when the solution manifold is unstable.
Limitations
- Analysis restricted to two-layer feedforward networks with scalar output and i.i.d. Gaussian inputs.
- Stability analysis is local; global nonlinear dynamics and accelerated convergence not fully characterized.
- Only presynaptic identity-based sign flips considered, not synapse-specific flips.
Experimental setup
Datasets: synthetic teacher-student data with i.i.d. Gaussian inputs
Metrics: test error Jacobian eigenvalues order parameter q convergence speed
Baselines: gradient descent
Builds on: gradient flow in linear networks feedback alignment Hebbian/anti-Hebbian similarity matching networks natural gradient descent
Closest papers by meaning
- Curly Flow Matching for Learning Non-gradient Field Dynamics (NeurIPS 2025 · similarity 0.86)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.85)
- Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network (ICML 2024 · similarity 0.85)
- Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron (NeurIPS 2024 · similarity 0.85)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.85)
- Identifying Equivalent Training Dynamics (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.