Research map / Architectures and efficiency / RNNs and normalization
Vector Quantization in the Brain: Grid-like Codes in World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Grid-like Code Quantization (GCQ), a VQ-style encoder-quantizer-decoder that compresses observation-action sequences jointly in space and time by matching latents against an action-conditioned codebook of CANN-generated bump attractors, yielding a world model for prediction, planning and inverse modeling.
Problem
Conventional VQ compresses static single observations, and world models typically use two-stage designs (VQ-VAE for space plus a separate temporal predictor), which makes long-horizon prediction unstable and planning costly.
Approach
Use a fixed codebook of CANN bump attractors on a torus, with per-action transitions (shifts) applied to base bumps, so that quantization is sequence-to-sequence template matching over observation-action trajectories; the codebook is fixed and only encoder/decoder are trained with reconstruction plus commitment loss and an STE.
Key results
- On Google Street View, GCQ (ViT, 112M params) reaches prediction FID 43.41 and PSNR 27.77 at 128x128, versus VQ+UNet and VQ+Transformer baselines that degrade over longer prediction horizons.
- At 80x40 resolution, the ViT variant achieves FIDr 13.27 and PSNRr 31.32 with 90M parameters, outperforming ResNet (330M, FIDr 48.05).
- Making the codebook learnable degrades prediction (FIDp 47.76 vs 43.41 fixed; PSNRp 24.48 vs 27.77).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViT encoder-decoder instead of ResNet (ablation) | GCQ · Google Street View (3×80×40) | FIDr: 48.05 -> 13.27 | Helped |
| Use ViT encoder-decoder instead of Hybrid ResNet-ViT (ablation) | GCQ · Google Street View (3×80×40) | FIDp: 22.31 -> 13.92 | Helped |
| Use ViT encoder-decoder instead of Hybrid ResNet-ViT (ablation) | GCQ · Google Street View (3×128×128) | FIDr: 41.55 -> 42.56 | Hurt |
| Use ViT encoder-decoder instead of Hybrid ResNet-ViT (ablation) | GCQ · Google Street View (3×128×128) | FIDp: 41.91 -> 43.41 | Hurt |
| Make codebook codes learnable instead of fixed (ablation) | GCQ · GSV | FIDp: 43.41 -> 47.76 | Hurt |
| Make codebook codes learnable instead of fixed (ablation) | GCQ · GSV | PSNRp: 27.77 -> 24.48 | Hurt |
| Use ViT rather than ResNet or Hybrid architecture (ablation) | GCQ · GSV | model parameter efficiency and overall performance | Mixed |
| Increase prediction horizon | GCQ versus VQ-based world models · Long-horizon prediction | prediction FID | Helped |
Surprising findings
- Learnable codebook entries hurt performance, which the authors attribute to training instability given the complex dynamic relationships among codes.
- In the static setting (sequence length 1), GCQ with a fixed codebook suffers only minimal degradation relative to VQ-VAE, and at 20 epochs outperforms VQ on ImageNet reconstruction.
Limitations
- Encoder and decoder process each sequence element independently rather than holistically over the sequence.
- Evaluated only on relatively simple, low-dimensional-action image datasets; scaling to larger, more diverse data is untested.
- Action space limited to 5^m combinations with 2D attractors, requiring design of the action mapping.
Experimental setup
Datasets: 2DMaze Google Street View MPI3D 3DShapes CIFAR-10 ImageNet
Benchmarks: 2DMaze Google Street View 3DShapes MPI3D ImageNet
Metrics: FID PSNR
Baselines: VQ+UNet VQ+Transformer VQ-VAE FSQ
Builds on: VQ-VAE continuous attractor neural networks LAPO TransDreamer
Closest papers by meaning
- Prediction and generalisation over directed actions by grid cells (ICLR 2021 · similarity 0.86)
- Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding (ICLR 2025 · similarity 0.86)
- AutoQVLA: Not All Channels Are Equal in Vision-Language-Action Model's Quantization (ICLR 2026 · similarity 0.86)
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.84)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.84)
- Learning Optimal Lattice Vector Quantizers for End-to-end Neural Image Compression (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.