Research map / Vision-language and multimodal / Multimodal LLMs

TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TPRU, a 24.75k-sample multi-image dataset with temporal reordering, next-frame prediction, and previous-frame review tasks, and trains Qwen2.5-VL models (3B/7B/32B) with GRPO to improve temporal and procedural understanding.

Problem

Small deployable multimodal LLMs struggle to understand temporal and procedural image sequences, and existing training data treats multi-image inputs as unordered collections or is evaluation-only.

Approach

Build a procedurally structured training set from embodied and GUI sources with three temporal tasks and hard negatives ("None of the choices"), then fine-tune Qwen2.5-VL with GRPO reinforcement learning to instill sequence reasoning in small models.

Key results

• TPRU-7B raises TPRU-Test accuracy from 50.33% to 75.70%, above GPT-4o.
• TPRU-32B reaches 68.42% on MuirBench, edging GPT-4o (68.00%); TPRU-7B reaches 65.04% overall on MuirBench.
• TPRU-7B on LEGO-Puzzles improves from 36.5% to 42.8%, with Ordering rising from 8.0% to 32.0%.

Surprising findings

• Removing the format reward only mildly reduces TPRU-Test accuracy (75.70% to 74.40%), suggesting gains are not from format mimicry.
• Video-specialized models such as LLaVA-Video-Qwen2-72B and Qwen2.5-Omni-7B score low on the MuirBench Ordering subtask (≤18.8%) despite general video pretraining.

Limitations

• TPRU-Test is small (461 instances) and manually curated, so variance and selection effects may matter.
• TPRU-25k relies on Qwen2.5-VL-72B for filtering and description generation, which introduces dependence on a teacher model.
• Gains on some sub-tasks (e.g., LEGO Multiview, Position for 32B) are negative or flat, and general benchmarks show mixed changes for the 32B model.

Experimental setup

Datasets: TPRU-25k ShareRobot GUI Odyssey Habitat LEGO instruction videos (Arvin Bricks) EPIC-KITCHENS

Benchmarks: TPRU-Test MuirBench LEGO-Puzzles MME-RealWorld-Lite BLINK RealWorldQA MMCR MMTBench MMStar MMMU-Dev

Metrics: accuracy

Baselines: GPT-4o Gemini-2.5-Flash Claude-3.5-Sonnet Qwen2.5-VL-72B InternVL3-78B LLaVA-Video-Qwen2-72B Qwen2.5-Omni-7B Long-VITA-16K SFT

Builds on: GRPO Qwen2.5-VL Easy-R1 LLaMA-Factory

Closest papers by meaning

• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
• BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.89)
• NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.89)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.89)
• RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.89)
• Large Language Models as Generalizable Policies for Embodied Tasks (ICLR 2024 · similarity 0.89)