Research map / Optimization / Optimizers
Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gauss-Southwell rather than uniform coordinate selection (ablation) | SRCD · Copying memory, early training | training loss | Helped |
| Use Gauss-Southwell rather than uniform coordinate selection (ablation) | SRCD · Copying memory, longer training | training loss | No clear effect |
| Update a block of coordinates instead of a single coordinate (ablation) | SRCD-GS · Copying memory | training loss | Helped |
| Update a block of coordinates instead of a single coordinate (ablation) | SRCD-GS · Copying memory | training loss relative to SRGD | Helped |
| Use SRCD-GS instead of SRGD | orthogonal RNN training · Copying memory | optim.step() runtime: 0.00197 -> 0.00177 | Helped |
| Use SRCD-GS instead of SRGD | orthogonal RNN training · Copying memory | loss.backward() + optim.step() runtime per iteration: 0.3311 -> 0.3191 | No clear effect |
Closest papers by meaning
- Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (ICML 2017 · similarity 0.89)
- Improved memory in recurrent neural networks with sequential non-normal dynamics (ICLR 2020 · similarity 0.88)
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (ICML 2019 · similarity 0.88)
- On orthogonality and learning recurrent networks with long term dependencies (ICML 2017 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.86)
- Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.