Research map / Multi-agent systems and games / LLM agents

SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SimuHome, a Matter-grounded, time-accelerated smart home simulator and a 600-episode benchmark across six query types (twelve categories with feasible/infeasible variants), evaluating 18 LLM agents and showing workflow scheduling is the hardest category.

Problem

Existing smart home agent benchmarks treat the home as static: they do not simulate how device actions change environmental variables over time, enforce operational dependencies, or evaluate scheduled workflows.

Approach

A tick-based, deterministic simulator grounded in Matter clusters that computes continuous environmental dynamics (temperature, humidity, illuminance, PM10) and supports time acceleration so scheduled workflows can be verified immediately; episodes are scored by simulator state checks or an LLM judge.

Key results

• GPT-5.1 achieves the best results on most categories, e.g., 100% on QT1-F and 60% on QT4-1-F, 72% on QT4-2-F, but averages 100+ seconds on demanding workflow tasks.
• Workflow scheduling (QT4) remains hardest; GPT-4.1 scores 50% on QT4-1-F but only 12% on QT4-1-IF with Gemini-2.5-Pro reaching 94% on QT4-1-IF.
• SFT on 204 GPT-5.1 trajectories improves infeasible detection by up to 26 points but gives limited gains on feasible workflow scheduling; recovery with oracle failure feedback reaches 55-67% on QT4.

Surprising findings

• Supervised fine-tuning on successful trajectories did not improve feasible coordinated scheduling (QT4-3-F) for either fine-tuned model.
• Reasoning models detect contradictions more reliably, yet on coordinated scheduling infeasible episodes GPT-5.1 (44%) underperforms Gemini-2.5-Pro (50%).

Limitations

• Simplified environmental update equations designed for agent evaluation rather than physical accuracy.
• Sim-to-real transfer to physical homes is not validated.
• Judge agreement (kappa 0.826) is substantial but not perfect; LLM judge used for infeasible episodes.

Experimental setup

Datasets: SimuHome episodes (600 generated)

Benchmarks: SimuHome benchmark (600 episodes, 12 categories)

Metrics: success rate average episode completion time

Baselines: HomeBench Sasha SAGE HiAgent

Builds on: ReAct HiAgent

Closest papers by meaning

• HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation (NeurIPS 2026 · similarity 0.90)
• TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning (NeurIPS 2026 · similarity 0.85)
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.85)
• VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications (ICLR 2026 · similarity 0.84)
• ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.84)
• Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments (ICLR 2026 · similarity 0.84)