Research map / Architectures and efficiency / RNNs and normalization
Differentiable optimization layers
Embeds optimization problems and efficient modules as differentiable end-to-end layers.
125 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “end · differentiable · translation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
31 papers in 2023–24 and 45 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 4.8% to 4.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Deep ADMM-Net for Compressive Sensing MRI (NeurIPS 2016 · 933 citations)
- 2017 · DSAC - Differentiable RANSAC for Camera Localization (CVPR 2017 · 637 citations)
- 2018 · Learning to Find Good Correspondences (CVPR 2018 · 570 citations)
- 2019 · SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction (CVPR 2019 · 558 citations)
- 2020 · Holistically-Attracted Wireframe Parsing (CVPR 2020 · 143 citations)
- 2021 · Hyperdimensional Computing as a Framework for Systematic Aggregation of Image Descriptors (CVPR 2021 · 55 citations)
- 2022 · Theseus: A Library for Differentiable Nonlinear Optimization (NeurIPS 2022 · 45 citations)
- 2023 · MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (ICLR 2023 · 69 citations)
- 2024 · PARA-Drive: Parallelized Architecture for Real-time Autonomous Driving (CVPR 2024 · 61 citations)
- 2025 · Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging (CVPR 2025 · 4 citations)
- 2026 · Identifiable Equivariant Networks are Layerwise Equivariant (ICML 2026 · 0 citations)
Earliest papers here
- Deep ADMM-Net for Compressive Sensing MRI (NeurIPS 2016 · 933 citations)
- Mnemonic Descent Method: A Recurrent Process Applied for End-To-End Face Alignment (CVPR 2016 · 367 citations)
- Sparse Coding for Third-Order Super-Symmetric Tensor Descriptors With Application to Texture Recognition (CVPR 2016 · 37 citations)
What has been tried
92 papers here report 997 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 7%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add ego vehicle states to PARA-Drive | nuScenes targeted scenarios · L2 error Aveall (m) | 0.9082 -> 0.7018 | Helped | PARA-Drive: Parallelized Architecture for Real-time Autonomous Driving (CVPR 2024 · 61 citations) |
| replace the neural SHPP solver with LKH-3 | TSP500 · Time | 19s -> 34m | Hurt | GLOP: Learning Global Partition and Local Construction for Solving Large-Scale Routing Problems in Real-Time (AAAI 2024 · 38 citations) |
| train with simulated noise to account for analog noise | Hand Gesture Recognition test set · accuracy | 68.9% -> 84.4% | Helped | PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors (CVPR 2024 · 12 citations) |
| replace the original Chamfer distance with CDR | Occ3D-nuScenes ablation · RayIoU | 29.2 -> 30.1 | Helped | OPUS: Occupancy Prediction Using a Sparse Set (NeurIPS 2024 · 10 citations) |
| Add row/column relation bias | SciTSR-COMP-A · F1-score | 84.9 -> 89.0 | Helped | Grab What You Need: Rethinking Complex Table Structure Recognition with Flexible Components Deliberation (AAAI 2024 · 9 citations) |
| Add ChanCA to the DIR | Set11, CR = 0.10 · PSNR | 31.42 -> 32.51; 1.09 dB | Helped | Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging (CVPR 2025 · 4 citations) |
| Add orthogonal loss | SU3 · AA@3° | 85.2 -> 86.1 | Helped | VPDETR: End-to-End Vanishing Point DEtection TRansformers (AAAI 2024 · 4 citations) |
| Dynamically update α and β instead of keeping α = 1 and β = 0.1 fixed | ActivityNet-v1.3 · Avg. mAP | 35.44% -> 36.27% | Helped | Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning (CVPR 2024 · 4 citations) |
| add feature alignment loss | Ablation configurations · ACC | 78.93 -> 86.79 | Helped | END^2: Robust Dual-Decoder Watermarking Framework Against Non-Differentiable Distortions (AAAI 2025 · 3 citations) |
| Add intra- and inter-instance attention | MapDR rule extraction · Recall (%) | 57.56 -> 71.75 | Helped | Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map (CVPR 2025 · 3 citations) |
| add neural network-predicted candidate points | adaptec05_2d · overflow | 10905 -> 431 | Helped | NeuralSteiner: Learning Steiner Tree for Overflow-avoiding Global Routing in Chip Design (NeurIPS 2024 · 2 citations) |
| add MLM without LAM | CIFAR-10 · test error | 14.08 -> 8.46(↓5.62) | Helped | MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network (AAAI 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 25 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 3 of 25 papers
- CIFAR-10 · 3 of 25 papers
- ImageNet-1K · 2 of 25 papers
- CIFAR100 · 2 of 25 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 76 papers; from OpenAlex and the papers' affiliations).
- Shanghai Jiao Tong University · 4 papers
- Huazhong University of Science and Technology · 3 papers
- King Abdullah University of Science and Technology · 3 papers
- Chinese Academy of Sciences · 3 papers
- Samsung · 2 papers
- Tsinghua University · 2 papers
- Tencent · 2 papers
- Peking University · 2 papers
- National University of Singapore · 2 papers
- University of Southern California · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Neural-symbolic reasoning (98 papers)
- Non-autoregressive translation (102 papers)
- Learned embeddings for retrieval (76 papers)
- Associative memory networks (48 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).