Research map / Architectures and efficiency / RNNs and normalization

Continuous Thought Machines

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Hebbian and predictive coding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Continuous Thought Machine, a recurrent architecture whose neurons each have private temporal models and whose latent representation is the synchronization of neuron activity over internal ticks, evaluated on mazes, ImageNet-1K, and parity.

Problem

Standard neural networks abstract away neuron-level temporal dynamics and use static snapshot representations, limiting sequential reasoning, adaptive computation, and biological plausibility.

Approach

Each neuron has private weights (neuron-level models) processing its own history of pre-activations over a decoupled internal time dimension, and pairwise neural synchronization (time-decayed inner products of post-activation histories) is used directly as the latent representation for attention queries and outputs, with a loss over the minimum-loss and maximum-certainty ticks enabling native adap

Key results

• On ImageNet-1K with 50 internal ticks and ResNet-152 backbone, the CTM reaches 72.47% top-1 and 89.89% top-5 accuracy (uncropped), explicitly not aiming for state-of-the-art.
• On 39x39 mazes the CTM outperforms feed-forward and LSTM baselines in accuracy on paths up to 100 steps and generalizes to 99x99 mazes via repeated re-application without positional embeddings.
• On ImageNet, a 0.8 certainty threshold lets most instances halt within fewer than 10 of 50 internal ticks, giving native adaptive compute.

Measured effects reported in the paper

Surprising findings

• LSTM baselines were unstable beyond a single layer or with more internal ticks, so 50 ticks performed better than 75 for them.
• The CTM barely used learnable synchronization decay for ImageNet but relied on it more for mazes.

Limitations

• Training times are extended because of the internal tick sequence.
• NLMs increase parameter counts relative to standard activations.
• Comparisons are limited in depth; results are preliminary and not state-of-the-art.

Experimental setup

Datasets: ImageNet-1K maze-dataset (39x39, 99x99, 19x19 mazes) CIFAR-10 CIFAR-100 MNIST

Benchmarks: ImageNet-1K 2D maze navigation (39x39, 99x99) cumulative parity (64-length) CIFAR-10 CIFAR-100

Metrics: top-1 accuracy top-5 accuracy calibration path accuracy

Baselines: LSTM feed-forward network Adaptive Computation Time PonderNet Liquid Time-Constant Networks

Builds on: Adaptive Computation Time PonderNet U-Net ResNet

Closest papers by meaning

• PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.86)
• Predictive Attractor Models (NeurIPS 2024 · similarity 0.86)
• Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.85)
• Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.85)
• Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.85)
• BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation (ICLR 2025 · similarity 0.85)