Research map / Vision-language and multimodal / Robot action and navigation

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ThinkJEPA, which pairs a dense-frame V-JEPA2 latent predictor with a uniformly sampled Qwen3-VL (Thinking) branch whose multi-layer features guide the predictor via FiLM, improving latent forecasting on EgoDex, EgoExo4D, BAIR and Physion.

Problem

Latent world models that predict from short dense observation windows miss long-horizon semantics, while VLMs alone are poor dense predictors due to sparse sampling, a language-output bottleneck, and small-data adaptation issues.

Approach

A dual-temporal design: a dense JEPA branch for fine-grained dynamics plus a VLM thinker on uniformly sampled frames; a hierarchical pyramid module aggregates multi-layer VLM features (ViT tokens and LLM hidden states) into FiLM modulation parameters for each JEPA predictor layer.

Key results

• On EgoDex, ThinkJEPA reaches ADE 0.061 and FDE 0.056 vs 0.071/0.066 for the V-JEPA Predictor and Acc 0.596 vs 0.471.
• On BAIR latent rollout, FD drops from 68.711 (V-JEPA Predictor) to 67.049 with ThinkJEPA.
• On Physion object-contact prediction, Acc rises from 0.741 to 0.754 and AUC from 0.821 to 0.828 over the V-JEPA Predictor.

Surprising findings

• Qwen3-VL prompt-only trajectory prediction is very poor (ADE 10.855), and the VLM-only trained baseline degrades sharply at longer rollout horizons.
• Single-source VLM guidance (encoder tokens only or AR tokens only) degrades results to near Qwen3-VL-only levels rather than helping the JEPA predictor.

Limitations

• Gains on Physion and BAIR are small (e.g., Physion Acc +0.013).
• Long-horizon rollout error still accumulates for ThinkJEPA as horizon grows (F@32 0.136).
• Ablation numbers in tables are inconsistent across variants (e.g., No-Th row matches V-JEPA Predictor while the ablation ThinkJEPA row differs from main results).

Experimental setup

Datasets: EgoDex EgoExo4D BAIR Robot Pushing Physion

Benchmarks: EgoDex EgoExo4D BAIR Robot Pushing Physion

Metrics: ADE FDE Acc FD SL1 CD AUC

Baselines: Qwen3-VL Thinking V-JEPA Predictor DINO-WM Persistence Decoder-only + Behavior Cloning Encoder-decoder + Flow Matching

Builds on: V-JEPA 2 FiLM DINO-WM

Closest papers by meaning

• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.88)
• World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.87)
• AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.87)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.86)