Research map / Optimization / Optimizers
SBO-RNN: Reformulating Recurrent Neural Networks via Stochastic Bilevel Optimization
Ziming Zhang, Yun Yue, Guojun Wu, Yanhua Li, Haichong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dense U instead of sparse U=beta*I (SGD) (ablation) | SBO-RNN · HAR-2 1% data | accuracy: 79.18 -> 90.25 | Improved |
| use dense U instead of sparse U=beta*I (SGD) (ablation) | SBO-RNN · HAR-2 10% data | accuracy: 90.76 -> 93.25 | Improved |
| sparse U (1.5K vs 17.9K parameters) reduces training time (SGD) (ablation) | SBO-RNN · HAR-2 | training time per batch: 0.60 -> 0.54 | Improved |
| replace SGD with heavy-ball momentum in lower level (dense) (ablation) | SBO-RNN · HAR-2 1% data | accuracy: 90.25 -> 86.59 | Hurt |
| replace SGD with Nesterov momentum in lower level (dense) (ablation) | SBO-RNN · HAR-2 10% data | accuracy: 93.25 -> 93.52 | No clear effect |
| replace SGD with momentum variants (ablation) | SBO-RNN · HAR-2 | training time per batch: 0.60 -> 0.67 | Hurt |
| increase training data from 1% to 10% for sparse SGD | SBO-RNN · HAR-2 | accuracy: 79.18 -> 90.76 | Improved |
| increase training data from 1% to 10% for dense SGD | SBO-RNN · HAR-2 | accuracy: 90.25 -> 93.25 | Improved |
| increase hidden dimension from 128 to 256 (ablation) | SBO-RNN · HAR-2 1% data | accuracy: 83.99% -> 86.29% | Improved |
| increase hidden dimension from 128 to 256 (ablation) | SBO-RNN · HAR-2 full data | accuracy: 95.08% -> 94.74% | Hurt |
| increase hidden dimension from 128 to 256 (ablation) | SBO-RNN · HAR-2 10% data | accuracy: 91.31% -> 92.37% | Improved |
| apply sparse-variant setup to SBO-RNN vs FastRNN | SBO-RNN vs FastRNN · HAR-2 full data | accuracy: 94.57% -> 94.96% | Improved |
Closest papers by meaning
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.89)
- Global optimality of Elman-type RNNs in the mean-field regime (ICML 2023 · similarity 0.88)
- Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians (NeurIPS 2020 · similarity 0.88)
- Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks (NeurIPS 2025 · similarity 0.88)
- Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis (ICLR 2024 · similarity 0.87)
- A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
