Research map / Image generation / Autoregressive and latent generation
Neural Language of Thought Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NLoTM, an unsupervised model combining a Semantic VQ-VAE (slot attention with block-level codebooks) and an autoregressive transformer prior over semantic concept tokens, for compositional discrete scene representation and generation.
Problem
Learning language-of-thought-like structured, discrete, compositional representations and probabilistic generative priors from unstructured visual observations, which patch-based VQ-VAEs and continuous object-centric models do not provide.
Approach
Factorizes each slot into M blocks, each quantized with its own shared codebook (SVQ), so object properties become reusable discrete codes; an autoregressive transformer prior (ALP) then samples these codes object by object to generate scenes.
Key results
- NLoTM achieves the lowest FID on CLEVR-Easy (32.50), CLEVR-Hard (43.12), and CLEVR-Tex (84.52), versus dVAE at 40.30, 65.89, and 112.80.
- On the odd-one-out OOD task, NLoTM Codebook reaches 99.1% OOD accuracy versus 67.6% for SysBinder and 24.0-55.6% for VQ-VAE and dVAE variants.
- On 2D Sprites with background, NLoTM reaches 42.19% generation accuracy versus 19.53% for VQ-VAE and 30.47% for dVAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic object-factor codes with the ALP prior instead of patch-based dVAE | NLoTM · CLEVR-Easy | FID: 40.30 -> 32.50 | Helped |
| Use semantic object-factor codes with the ALP prior instead of patch-based dVAE | NLoTM · CLEVR-Hard | FID: 65.89 -> 43.12 | Helped |
| Use semantic object-factor codes with the ALP prior instead of patch-based dVAE | NLoTM · CLEVR-Tex | FID: 112.80 -> 84.52 | Helped |
| Use semantic object-factor codes rather than patch codes | NLoTM · 2D Sprites (4 obj) | Generation Accuracy (in %): 62.50 -> 66.41 | Helped |
| Use semantic object-factor codes rather than patch codes | NLoTM · 2D Sprites w/ bg (4 obj) | Generation Accuracy (in %): 30.47 -> 42.19 | Helped |
| Use semantic object-factor codes rather than patch codes | NLoTM · 2D Sprites w/ bg (4 obj) | FID: 58.14 -> 58.50 | Hurt |
| Use NLoTM codebook prototype vectors rather than SysBinder continuous representations | 2D Sprites odd-one-out · OOD | OOD accuracy (%): 67.6 -> 99.1 | Helped |
| Use codebook prototype vectors rather than codebook indices | NLoTM · 2D Sprites odd-one-out | OOD accuracy (%): 46.8 -> 99.1 | Helped |
| Use codebook prototype vectors rather than SysBinder continuous representations | CLEVR-Hard Property Comparison · OOD | OOD accuracy (%): 70.09 -> 71.15 | No clear effect |
| Increase dVAE prior capacity from 8 to 16 layers (ablation) | dVAE · CLEVR-Hard | FID: 65.89 -> 60.75 | Helped |
| Increase SVQ blocks from 1 to 4 (ablation) | NLoTM · 2D Sprites (3 obj) | Generation Accuracy (in %): 0.00 -> 75.78 | Helped |
| Increase SVQ blocks from 8 to 16 (ablation) | NLoTM · 2D Sprites (3 obj) | Generation Accuracy (in %): 75.00 -> 55.47 | Hurt |
Surprising findings
- VQ-VAE achieves the lowest FID on 2D Sprites with background (58.14) yet only 19.53% generation accuracy, showing FID can disagree with structural correctness.
- On CLEVR-Hard property comparison, SysBinder (79.60% ID) slightly beats NLoTM Codebook (75.86% ID), although NLoTM is better OOD.
Limitations
- Evaluated only on synthetic, visually simple datasets; scaling to realistic scenes is not shown beyond a small Google Scanned Objects experiment.
- All latent representations are discrete, so continuous factors such as position and pose are not modeled well.
- Generation accuracy decreases with too many blocks or large codebooks, likely requiring higher-capacity priors.
Experimental setup
Datasets: 2D Sprites CLEVR-Easy CLEVR-Hard CLEVR-Tex Google Scanned Objects
Benchmarks: 2D Sprites CLEVR-Easy CLEVR-Hard CLEVR-Tex Odd-One-Out (2D Sprites) CLEVR-Hard Property Comparison Google Scanned Objects
Metrics: FID generation accuracy steps to 98% ID accuracy OOD accuracy FG-ARI
Baselines: VQ-VAE with PixelCNN prior dVAE with transformer prior GENESIS-v2 SysBinder Slot Attention SLATE
Builds on: VQ-VAE Slot Attention SysBinder dVAE PixelCNN
Closest papers by meaning
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.86)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.85)
- Concept-Guided Interpretability via Neural Chunking (NeurIPS 2025 · similarity 0.85)
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.85)
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.85)
- LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.