Research map / Architectures and efficiency / RNNs and normalization
Neural Networks and the Chomsky Hierarchy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a stack (Stack-RNN) to the RNN controller | RNN · Regular-to-DCF tasks (Stack Manipulation, Reverse String, Modular Arithmetic) | score (%): Stack Manipulation 56.0 to 100.0; Reverse String 62.0 to 100.0; Modular Arithmetic 41.3 to 96.1 | Helped |
| Add a tape (Tape-RNN) to the RNN controller | RNN · Context-sensitive tasks (Duplicate String, Binary Addition) | score (%): Duplicate String 50.3 to 100.0; Binary Addition 50.3 to 100.0 | Helped |
| Use Transformer encoder (best positional encoding) instead of RNN | Transformer · Regular tasks (Parity Check, Modular Arithmetic Simple, Cycle Navigation) | score (%): Parity Check 52.0, Modular Arithmetic (Simple) 24.2 vs RNN/LSTM 100.0 | Hurt |
| Use Transformer with positional encodings on Reverse String | Transformer · Reverse String (DCF) | score (%): 62.3 vs RNN 62.0 and LSTM 60.9 | No clear effect |
| Use LSTM instead of RNN controller | LSTM · Bucket Sort (CS) | score (%): 99.3 vs RNN 27.9 | Helped |
| Add a stack to the RNN controller (Stack-RNN) on Bucket Sort | Stack-RNN · Bucket Sort (CS) | score (%): 78.1 vs RNN 27.9 | Helped |
| Increase training data and model size for Transformers (scaling experiments) (ablation) | Transformer · Context-sensitive and higher-level tasks | score (%) | No clear effect |
| Use Transformer with positional encodings on regular tasks, distinct from permutation-invariant case | Transformer · Even Pairs (Regular) | score (%): Even Pairs 96.4 solved, but Parity Check 52.0 and Cycle Navigation 61.9 fail | Mixed |
Closest papers by meaning
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.88)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.88)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.86)
- On the creation of narrow AI: hierarchy and nonlocality of neural network skills (NeurIPS 2025 · similarity 0.86)
- Algorithm Development in Neural Networks: Insights from the Streaming Parity Task (ICML 2025 · similarity 0.86)
- Task Generalization with Autoregressive Compositional Structure: Can Learning from $D$ Tasks Generalize to $D^T$ Tasks? (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.