Research map / Large language models / Agents, code and math

R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner

CVPR 2025 · Large language models › Agents, code and math · approach: Long-horizon LLM agents · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Maps the robot's room into a grid chessboard so LLMs (GPT-4 zero-shot, fine-tuned Mistral-7B and Llama-7B) can plan low-level single-step moves, trained with a four-part chain-of-thought decision task.

Problem

LLMs are used mainly as high-level planners for embodied tasks because they cannot ground spatial perception well enough to generate precise low-level actions; existing LLM agents depend on external navigation controllers.

Approach

Abstract the egocentric semantic map into a compact grid chessboard with object occupancy coordinates, let the LLM act as a chess player predicting the next adjacent grid cell, and fine-tune it jointly on high-level task decomposition and chain-of-thought spatial decision subtasks (key info extraction, direction judgment, target prediction, selection analysis).

Key results

• R2C-Mistral-7B reaches 22.31% SR / 32.40% GC on ALFRED Val Seen and 22.35% / 31.97% on Val Unseen, comparable to GPT-4-based R2C and above LLM-P (GPT).
• With ground-truth segmentation and goals, R2C-Mistral-7B reaches 48.18% SR on Val Seen and 53.33% on Val Unseen.
• On AI2THOR ObjectNav, R2C-Mistral-7B (without extra fine-tuning) gets 52.25% SR and 29.56% SPL versus 46.16% SR for S2P.

Measured effects reported in the paper

Surprising findings

• Feeding the chessboard image directly to GPT-4o performs worse than textual coordinate input to GPT-4 (43.3% vs 68.3% overall SR on open-vocabulary tasks).
• Removing CoT-D reasoning causes a large drop, especially on unseen scenes (from 53.33 to 41.96 SR on Val Unseen with GT segmentation and goals).

Limitations

• Performance is bottlenecked by segmentation quality and ambiguous instructions, as shown by large gains with ground-truth inputs.
• Long-horizon tasks such as Heat & Place degrade due to error accumulation.
• Struggles with very large scenes; the one-cell-per-step action space limits efficiency.

Experimental setup

Datasets: ALFRED

Benchmarks: ALFRED AI2THOR ObjectNav open-vocabulary tasks (60 GPT-4-generated trials)

Metrics: success rate goal-conditioned success rate SPL

Baselines: E.T. HiTUT M-TRACK FILM LEBP SayCan LLM-P (GPT) SAVN S2P

Builds on: SayCan LLM-P FILM chain-of-thought prompting

Closest papers by meaning

• Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.90)
• Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.90)
• OntoPlan: An Ontology-Grounded Scene Representation and Agentic Framework for Scalable Robot Task Planning (NeurIPS 2026 · similarity 0.89)
• Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.89)
• Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.89)
• RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.89)