Research map / Architectures and efficiency / RNNs and normalization
Neural-symbolic reasoning
Combines recurrent networks and differentiable optimization with symbolic reasoning and program synthesis.
98 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “end · differentiable · translation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
23 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.5% to 1.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Meta-Learning with Memory-Augmented Neural Networks (ICML 2016 · 1,224 citations)
- 2017 · The Predictron: End-To-End Learning and Planning (ICML 2017 · 150 citations)
- 2018 · Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-Offs by Selective Execution (AAAI 2018 · 190 citations)
- 2019 · Attention, Learn to Solve Routing Problems! (ICLR 2019 · 186 citations)
- 2020 · Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (ICLR 2020 · 58 citations)
- 2021 · Towards Impartial Multi-task Learning (ICLR 2021 · 26 citations)
- 2022 · Neuro-Symbolic Inductive Logic Programming with Logical Neural Networks (AAAI 2022 · 42 citations)
- 2023 · End-to-End Learning for Optimization via Constraint-Enforcing Approximators (AAAI 2023 · 12 citations)
- 2024 · End-to-End Learning of LTLf Formulae by Faithful LTLf Encoding (AAAI 2024 · 8 citations)
- 2025 · Nested Learning: The Illusion of Deep Learning Architectures (NeurIPS 2025 · 9 citations)
- 2026 · Rethink the Role of Neural Decoders in Quantum Error Correction (ICML 2026 · 0 citations)
Earliest papers here
- Meta-Learning with Memory-Augmented Neural Networks (ICML 2016 · 1,224 citations)
- Visual Learning of Arithmetic Operation (AAAI 2016 · 11 citations)
- The Predictron: End-To-End Learning and Planning (ICML 2017 · 150 citations)
What has been tried
77 papers here report 781 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 10%, hurt 21%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase encoding size L | kf=9 · accuracy | Helped | End-to-End Learning of LTLf Formulae by Faithful LTLf Encoding (AAAI 2024 · 8 citations) | |
| Predict state-values instead of action-values | 1K puzzles (ablation setup) · Puzzle accuracy (%) | 83.3 -> 77.5 | Hurt | Amortized Planning with Large-Scale Transformers: A Case Study on Chess (NeurIPS 2024 · 8 citations) |
| Add the input projection step | Varying dimension and number of bound vector pairs · retrieval noise | 19.38 -> 7.18 | Helped | A Walsh Hadamard Derived Linear Vector Symbolic Architecture (NeurIPS 2024 · 5 citations) |
| Add downstream-layer reconstruction loss to end-to-end KL and sparsity training | GPT2-small residual stream before attention layer 6 · downstream reconstruction loss | Helped | Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning (NeurIPS 2024 · 4 citations) | |
| Add end-to-end classifier gradients to guide SIREN meta-learning | CIFAR-10 validation · accuracy | 38.98 -> 55.79 | Helped | End-to-End Implicit Neural Representations for Classification (CVPR 2025 · 3 citations) |
| Remove the codebook | SAR · accuracy | 99.27 -> 89.02 | Hurt | Discrete Dictionary-based Decomposition Layer for Structured Representation Learning (NeurIPS 2024 · 1 citation) |
| remove MI loss | IOI · Logit Difference | Hurt | IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck (ICML 2025 · 0 citations) | |
| Use a fully trainable mask independent of the code | (31,16) and (32,11), Eb/N0=4 · -ln(BER) | 5.16 -> 4.83; 4.40 -> 4.04 | Hurt | Learning Linear Block Error Correction Codes (ICML 2024 · 0 citations) |
| swap the left and right embeddings | S5 · Mean Accuracy | 99.99% -> 1% | Hurt | Grokking Group Multiplication with Cosets (ICML 2024 · 0 citations) |
| Use dynamic rather than static embeddings | SR(U(10, 40)) unsat test · proven (%) | 14.72 -> 37.33; 25.38 -> 95.2; 28.72 -> 60.5 | Helped | Learning Better Representations From Less Data For Propositional Satisfiability (NeurIPS 2024 · 0 citations) |
| add balanced weighting to the loss | 8-task GLUE benchmark · average normalized score | 77.4 -> 85.3 | Helped | Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 3 of 10 papers
- CIFAR-10 · 2 of 10 papers
- SVHN · 2 of 10 papers
- DTD · 2 of 10 papers
Baselines
- MLP · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 40 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 3 papers
- Google DeepMind · 2 papers
- Meta · 2 papers
- Cornell University · 2 papers
- Johns Hopkins University · 2 papers
- Microsoft · 2 papers
- Booz Allen Hamilton · 2 papers
- Google · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Differentiable optimization layers (125 papers)
- Non-autoregressive translation (102 papers)
- Learned embeddings for retrieval (76 papers)
- Associative memory networks (48 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).