Research map / Vision-language and multimodal / Multimodal LLMs

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LMAct, an open benchmark testing whether frontier multimodal LMs can learn to act in interactive tasks from up to 512 in-context expert demonstration episodes (up to 1M tokens), finding that performance often fails to reach expert level and rarely improves with more demonstrations.

Problem

Frontier multimodal LMs struggle with simple interactive decision-making; it is unclear whether conditioning on large numbers of multimodal expert demonstrations in context can unlock their decision-making ability.

Approach

A unified zero-, few-, and many-shot in-context imitation learning benchmark with long multimodal (ASCII/FEN/PGN/RGB) demonstrations across six simple fully observable environments, evaluating closed frontier models as policies with expert demonstrations from Stockfish, minimax, Gato data, and solvers.

Key results

• Models rarely reach expert performance; o1 models achieve near-optimal tic-tac-toe and o1-preview/o1 almost solve crosswords.
• Atari Phoenix: no model outperforms the random action baseline; Gemini 1.5 Flash performs best.
• Most models show performance largely independent of number of demonstrations; Gemini models improve with more demos on grid world coordinate and RGB observations.
• In the replay control, all models except o1-mini perform well at copying the single demonstration episode.

Measured effects reported in the paper

Surprising findings

• Performance is mostly independent of the number of demonstrations, and often degrades with more (in-context interference on cheetah run and ASCII grid world).
• Models often output illegal chess moves even when legal actions are listed in the prompt.

Limitations

• Closed-source APIs with models that may be updated, limiting reproducibility.
• Results are a lower bound since prompt variations were limited.
• Environments are simple, fully observable and may not transfer to real-world tasks.

Experimental setup

Datasets: Gato training data (Atari Phoenix, DM Control cheetah run) Stockfish-generated chess demonstrations genxword crosswords with Ginsberg clue list Encyclopedia of Chess Openings

Benchmarks: LMAct Atari Phoenix chess vs Stockfish level 0 7x7 crosswords DM Control cheetah run grid world navigation tic-tac-toe vs random

Metrics: average episode return percentage of illegal actions action replay accuracy

Baselines: random legal action policy expert policy BALROG

Builds on: Gato BALROG chain-of-thought prompting Stockfish

Closest papers by meaning

• VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought (NeurIPS 2024 · similarity 0.88)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.87)
• Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.87)
• Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning (CVPR 2025 · similarity 0.87)
• Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.86)
• Policy Improvement using Language Feedback Models (NeurIPS 2024 · similarity 0.86)