Research map / Optimization / Optimizers
Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians
Rainer Engelken
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gradient flossing before training | Vanilla RNN · Delayed copy task | max reliably trainable delay: 20 -> 40 | Improved |
| add gradient flossing during training in addition to before (ablation) | Vanilla RNN · Delayed temporal XOR d=70 | test accuracy: chance level -> >80% | Improved |
| add preflossing only (no flossing during training) (ablation) | Vanilla RNN · Delayed temporal XOR | test accuracy: chance at d>=40 -> chance at d=70 | Mixed |
| flossing during training on 3-input XOR (ablation) | Vanilla RNN · 3-input XOR task | max solvable delay: preflossing fails for d>60 | Improved |
| continue gradient flossing throughout all training epochs (ablation) | Vanilla RNN · temporal XOR | test accuracy | Hurt |
| regularize spectral radius or one-step Jacobians instead of Lyapunov exponents (ablation) | Vanilla RNN · delayed tasks | trainability | No clear effect |
| increase number of flossed Lyapunov exponents k (ablation) | Vanilla RNN · binary delayed XOR | test accuracy | Improved |
| flossing all N exponents before training only (ablation) | Vanilla RNN · hardest XOR tasks | test accuracy | No clear effect |
| add gradient flossing | Vanilla RNN · long-term Jacobian | condition number | Improved |
Closest papers by meaning
- On the difficulty of learning chaotic dynamics with RNNs (NeurIPS 2022 · similarity 0.87)
- Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.86)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.85)
- Stabilizing Backpropagation Through Time to Learn Complex Physics (ICLR 2024 · similarity 0.85)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
