Research map / Continual learning / Online and replay-based
Locality Sensitive Sparse Encoding for Learning World Models Online
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Losse-FTL, a linear world model on a locality-sensitive sparse random-feature encoding updated incrementally in closed form (Follow-The-Leader), enabling online Dyna MBRL without replay or forgetting.
Problem
Learning accurate world models online for model-based RL under nonstationary data, where neural world models suffer catastrophic forgetting and need costly full re-training to approximate Follow-The-Leader.
Approach
Combine a linear regressor with a high-dimensional sparse random feature encoder (random projection plus soft binning, Losse) so that the FTL least-squares solution can be updated incrementally, touching only the small block of weights activated by each sample.
Key results
- On MNIST denoising with patch size 9, Losse achieves MSE 0.28 (x10^-1) versus 0.32 for Fourier, 0.35 for ReLU, 0.36 for Tile Code and 0.36 for NN.
- On the piecewise random walk stream, Losse-FTL keeps very low error across all correlation levels while online and batch NNs degrade as correlation increases.
- In Dyna MBRL on Gym Mujoco, Losse-FTL world models perform on par with or better than Full-replay NN world models, while Online NN fails and SI and Coreset are inferior to Full-replay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Losse features instead of Fourier, ReLU, or Tile Code features | Linear model for MNIST image denoising · MNIST image denoising, patch sizes 9, 16, 25, 36, and 49 | mean squared error: patch size 9: Fourier 0.32, ReLU 0.35, Tile Code 0.36 -> Losse 0.28; patch size 16: ReLU 0.34 -> Losse 0.29; patch size | Helped |
| Use Losse features instead of NN | Image denoising model · MNIST image denoising, patch sizes 9, 16, and 25 | mean squared error: patch size 9: 0.36 -> 0.28; patch size 16: 0.35 -> 0.29; patch size 25: 0.34 -> 0.31 | Helped |
| Use Losse features instead of NN | Image denoising model · MNIST image denoising, patch size 49 | mean squared error: 0.33 -> 0.40 | Hurt |
| Learn online with Losse-FTL instead of online neural-network training | Supervised stream learner · Piecewise random walk stream, across correlation levels | mean squared error | Helped |
| Train neural networks online under covariate shift rather than using Losse-FTL | Supervised stream learner · Piecewise random walk stream, higher correlation levels | mean squared error | Hurt |
| Use Losse-FTL world models instead of NN world models with replay | Dyna agent · Gridworld | episode return | Helped |
| Use Losse-FTL world models instead of the compared baselines | Dyna agent · Mountain Car and Acrobot | episode return | Helped |
| Use Losse-FTL instead of Full-replay world models | Dyna agent · Gym MuJoCo continuous-control tasks | episode return | Mixed |
| Train NN world models online rather than using the Model-free agent | Dyna agent · Gym MuJoCo continuous-control tasks | episode return | Hurt |
| Add Synaptic Intelligence or Coreset replay to online NN world models | Dyna agent · Gym MuJoCo continuous-control tasks | episode return | Helped |
| Increase Coreset replay size (ablation) | Coreset world model · Gridworld | episode return | Helped |
| Increase Coreset replay size (ablation) | Coreset world model · Gridworld | wall-clock efficiency | Hurt |
Surprising findings
- Gradient-descent updates on sparse Losse features still suffer catastrophic forgetting until very high sparsity (lambda=30, about 9000-dimensional features with 40 nonzeros), so sparsity alone does not prevent forgetting.
- Online NN world models fail even on Gym Mujoco tasks, performing worse than the model-free baseline.
Limitations
- Not validated on very high-dimensional state spaces such as Humanoid.
- Not validated on tasks with image observations.
- Linear models on fixed random features may struggle with high-dimensional problems.
Experimental setup
Datasets: MNIST Gridworld Mountain Car Acrobot Gym Mujoco piecewise random walk synthetic stream
Benchmarks: MNIST image denoising piecewise random walk stream learning Gridworld Mountain Car Acrobot Gym Mujoco
Metrics: mean squared error normalized episode return wall-clock time
Baselines: Full-replay Online NN Synaptic Intelligence Coreset (reservoir sampling) Random Fourier Features Random ReLU Features Random Tile Coding Model-free DQN/SAC
Builds on: Dyna MBPO Locality Sensitive Hashing Least-Squares TD analytic class-incremental learning
Closest papers by meaning
- Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.87)
- Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL (NeurIPS 2026 · similarity 0.87)
- RLVR-World: Training World Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.86)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.86)
- Efficient World Models with Context-Aware Tokenization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.