Research map / Multi-agent systems and games / Human-AI and goal-conditioned
COOPERA: Continual Open-Ended Human-Robot Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COOPERA, a framework simulating LLM-driven, trait-based humans over multi-day household routines, plus a benchmark and VLM-plus-classifier robot that personalizes assistance from end-of-day feedback.
Problem
Existing human-robot collaboration benchmarks are episodic with predefined tasks and lack realistic long-term human models, so robots cannot learn individual traits and time-dependent intentions over days.
Approach
Simulate humans with Big-5 traits and LLM-generated whole-day intentions and tasks (using memory retrieval, search, and reflexion) executed with whole-body motion in Habitat 3.0; the robot decouples inference into intention and task stages, using a VLM to propose candidates and LoRA-finetuned classifiers, optimized daily from human feedback.
Key results
- Intention classifier reaches 0.995 accuracy and task classifier 0.830 accuracy on distinguishing simulated humans.
- Main method achieves highest within-day and across-day improvement, second only to Oracle; in Setting 4 success rate 0.344 vs. ablation No Traits 0.206.
- Simulated intentions reach SBERT similarity 0.810 with real-human intentions versus about 0.52-0.55 for generic and mismatched baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the aligned human profile when generating simulated intentions | COOPERA human simulation · Real-human intention alignment | SBERT similarity: 0.554 -> 0.810; 0.523 -> 0.810 | Helped |
| use the aligned human profile when generating simulated intentions | COOPERA human simulation · Real-human intention alignment | OpenAI embedding similarity: 0.537 -> 0.772; 0.543 -> 0.772 | Helped |
| fine-tune on prior scenes before collaborating in an unseen scene | COOPERA · Unseen scene, same human | predicate-based success rate: 0.269 -> 0.465 | Helped |
| fine-tune on other humans before collaborating with a new human | COOPERA · Unseen human | predicate-based success rate: 0.258 -> 0.343 | Helped |
| remove simulated conversations and profile extension (ablation) | COOPERA human simulation · Human classification | intention accuracy: 0.950 -> 0.995 | Hurt |
| remove simulated conversations and profile extension (ablation) | COOPERA human simulation · Human classification | task accuracy: 0.800 -> 0.830 | Hurt |
| propose all daily intentions at once, without hourly history and Reflexion (ablation) | COOPERA human simulation · Temporal dependence prediction | accuracy: 0.751 -> 0.789 | Hurt |
| propose all daily intentions at once, without hourly history and Reflexion (ablation) | COOPERA human simulation · Temporal dependence prediction | F1: 0.740 -> 0.790 | Hurt |
| remove human trait inference (ablation) | COOPERA assistive agent · Settings 3 and 4: multiple humans | predicate-based success rate: Setting 3: 0.239 -> 0.439; Setting 4: 0.206 -> 0.344 | Hurt |
| remove temporal context learning (ablation) | COOPERA assistive agent · Settings 1–4 | predicate-based success rate: Setting 1: 0.452 -> 0.505; Setting 2: 0.414 -> 0.465; Setting 3: 0.408 -> 0.439; Setting 4: 0.299 -> 0.344 | Hurt |
| replace Llama-3.2-11B robot-VLM backbone with LLaVA-1.6-Mistral-7B (ablation) | COOPERA assistive agent · Settings 1–4 | predicate-based success rate: Setting 1: 0.487 -> 0.505; Setting 2: 0.424 -> 0.465; Setting 3: 0.362 -> 0.439; Setting 4: 0.310 -> 0.344 | Hurt |
Surprising findings
- Offline and human-in-the-loop real participants achieved comparable or sometimes higher success than simulated humans, attributed to participants following more consistent recalled routines.
- Generalizing to a new human was much harder than generalizing to a new scene (0.343 vs 0.465 finetuned success).
Limitations
- Focuses on single-human settings; multi-human collaboration left for future work.
- Evaluations are primarily in simulation, with real-world validation limited to real-human routines and interactions.
- Real-human human-in-the-loop experiments use text input instead of direct control of the simulated agent.
Experimental setup
Datasets: SPC: Synthetic-Persona-Chat HSSD YCB Dataset Motion-X AMASS
Benchmarks: COOPERA (Watch-and-Help-based open-ended HRC settings)
Metrics: F1-based success rate predicate-based success rate accuracy Pearson correlation SBERT similarity L1 correlation with human verification
Baselines: Direct Prompting Direct Finetuning Oracle Random Intention Agnostic Human & Context Agnostic
Builds on: Watch-and-Help Habitat 3.0 Reflexion LoRA
Closest papers by meaning
- Towards Better Robot Learners: Leveraging Implicit and Explicit Human Feedback Together in Human Robot Interactions (AAAI 2025 · similarity 0.86)
- Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots (ICLR 2024 · similarity 0.85)
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.85)
- Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators (NeurIPS 2024 · similarity 0.84)
- Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems (NeurIPS 2024 · similarity 0.84)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.