Research map / Architectures and efficiency / RNNs and normalization
NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuralSolver, a convolutional-LSTM recurrent solver with a max-pooling processing module and curriculum training that extrapolates learned algorithms from small to very large inputs in both same-size and different-size tasks.
Problem
Recurrent solvers that learn algorithms on small inputs and run them on larger ones have only been applied to same-size tasks (input and output the same size), leaving different-size tasks (e.g., classification or decision-making over arbitrary-size images) unaddressed.
Approach
A layer-normalized convolutional LSTM that re-consumes the original input at each iteration, followed by a convolutional processing head with global max-pooling as an aggregation step, trained with a size curriculum; this makes recurrent extrapolation work for both same-size and different-size tasks without a progressive loss.
Key results
- On same-size tasks NeuralSolver reaches 100.00% on Prefix-Sum (32 to 512) and Maze (24 to 124) and 84.30% on Chess, versus 99.78% / 86.63% / 79.99% for Bansal et al.
- On different-size tasks NeuralSolver reaches 100% on 1S-Maze, GoTo, Pong and Doorkey, while Bansal et al. gets 74.14, 64.32, 71.97 and 97.13%.
- NeuralSolver uses 0.23M parameters for different-size tasks, a 92% reduction vs. Bansal et al. (3.15M) and 99.6% vs. FeedForward (71.57M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeuralSolver instead of Bansal et al. | recurrent solvers · Thin-Maze (11→61) | accuracy: 46.78 -> 99.97 | Helped |
| replace global max pooling with average pooling (ablation) | NeuralSolver · DoorKey | accuracy: 100.00 -> 51.86 | Hurt |
| use five convolutional layers instead of one in the recurrent module (ablation) | NeuralSolver · GoTo | accuracy: 100.00 -> 84.16 | Hurt |
| remove curriculum learning (ablation) | NeuralSolver · GoTo | accuracy: 100.00 -> 64.12 | Hurt |
| remove curriculum learning (ablation) | NeuralSolver · Pong | accuracy: 100.00 -> 100.00 | No clear effect |
| replace the LSTM with a ResNet block (ablation) | NeuralSolver · 1S-Maze | accuracy: 100.00 -> 81.87 | Hurt |
| replace the LSTM with a ResNet block (ablation) | NeuralSolver · GoTo | accuracy: 100.00 -> 67.28 | Hurt |
| reduce model width from 64 to 48 (ablation) | NeuralSolver · 1S-Maze | accuracy: 100.00 -> 98.79 | Hurt |
| reduce model width from 64 to 32 (ablation) | NeuralSolver · GoTo | accuracy: 100.00 -> 96.56 | Hurt |
| reduce model width from 64 to 24 (ablation) | NeuralSolver · DoorKey | accuracy: 100.00 -> 91.21 | Hurt |
| use NeuralSolver instead of Bansal et al. | recurrent solvers · GoTo ([6,20]→128) | accuracy: 64.32 -> 100.00 | Helped |
| use NeuralSolver instead of Bansal et al. | recurrent solvers · Doorkey 128×128 | average reward ×10²: 24.71 -> 98.02 | Helped |
Surprising findings
- Removing the progressive loss from the baseline's training is not needed: NeuralSolver avoids overthinking without any progressive loss.
- Using a single convolutional layer in the recurrent block outperformed a five-layer version, despite requiring more iterations.
Limitations
- Training efficiency gap to upper-bound performance remains when trained only on very small problems in different-size tasks.
- Evaluated on synthetic grid and maze tasks with fixed-output classification; not on natural image or language tasks.
Experimental setup
Datasets: Prefix-Sum Maze Thin-Maze Chess GoTo 1S-Maze Pong Doorkey (Minigrid-inspired)
Benchmarks: Prefix-Sum Maze Thin-Maze Chess 1S-Maze GoTo Pong Doorkey Minigrid Doorkey reward evaluation
Metrics: best accuracy over iterations (mean and std over 10 seeds) almost stochastic order test average reward parameter count GigaMACs
Baselines: Bansal et al. recurrent solver Bansal et al. + progressive loss FeedForward ResNet Random GRU/LocRNN recurrent modules
Builds on: Recurrent Equilibrium / recurrent ResNet solver (Bansal et al.) progressive loss recall module convolutional LSTM Gal dropout
Closest papers by meaning
- Breakeven complexity: A new perspective on neural partial differential equation solvers (NeurIPS 2026 · similarity 0.85)
- Linear Transformers Implicitly Discover Unified Numerical Algorithms (NeurIPS 2025 · similarity 0.85)
- Fast T2T: Optimization Consistency Speeds Up Diffusion-Based Training-to-Testing Solving for Combinatorial Optimization (NeurIPS 2024 · similarity 0.84)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.84)
- PRDP: Progressively Refined Differentiable Physics (ICLR 2025 · similarity 0.84)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.