Research map / 3D vision / 3D generation
FAVOR: Full-Body AR-Driven Virtual Object Rearrangement Guided by Instruction Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FAVOR, a 3k-sequence (7.17M frame) full-body rearrangement motion dataset collected with MoCap and AR glasses, and FAVORITE, an instruction-guided pipeline that grounds objects with GPT-4 and Owl-ViT, generates grasp keyframes with a cVAE, and fills motion with an in-betweening network.
Problem
Generating natural full-body human motion for rearranging objects in complex scenes guided by text instructions is under-explored, because existing human-object interaction datasets cover only simple grasp-and-use interactions and lack paired instruction-motion data.
Approach
Collect a large instruction-paired rearrangement dataset by combining MoCap with AR glasses that render virtual scenes, then build a staged pipeline: LLM/VLM-based object grounding, cVAE keyframe grasp-pose generation, and auto-regressive in-betweening with SDF-based penetration and obstacle losses.
Key results
- Perceptual study: FAVOR ground truth averages 4.62 vs. generated motion 3.85 on a 1-5 Likert scale.
- Grasp quality: KNET G achieves contact ratio 0.99 and SIV 7.05 versus FAVOR's 0.99 and 0.83.
- FLEX-based grasp poses led to nearly half of attempts being unsuccessful, while KNET grasps were stable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ground-truth keyframe poses instead of KNET-predicted poses (ablation) | INET 0:G · Motion generation | PSKL-J (g.t., p.): 0.255 -> 0.237 | Helped |
| Use ground-truth keyframe poses instead of KNET-predicted poses (ablation) | INET 0:G · Motion generation | PSKL-J (p., g.t.): 0.209 -> 0.193 | Helped |
| Use ground-truth keyframe poses instead of KNET-predicted poses (ablation) | INET G:P · Motion generation | PSKL-J (g.t., p.): 0.706 -> 0.491 | Helped |
| Use ground-truth keyframe poses instead of KNET-predicted poses (ablation) | INET G:P · Motion generation | PSKL-J (p., g.t.): 0.657 -> 0.511 | Helped |
| Add object optimization during grasping (ablation) | INET · Grasping sequences | CR: 0.96 -> 0.89 | Helped |
| Add object optimization during grasping (ablation) | INET · Grasping sequences | SIV: 11.81 -> 6.50 | Helped |
| Replace FLEX grasp poses with KNET-predicted poses (ablation) | INET · Grasping sequences | CR: 0.50 -> 0.89 | Helped |
| Replace FLEX grasp poses with KNET-predicted poses (ablation) | INET · Grasping sequences | SIV: 6.23 -> 6.50 | Mixed |
Surprising findings
- Placement keyframes (T^P) yield less stable grasps than grasping keyframes, suggesting placement is harder than grasping.
- AR-based collection greatly speeds recording (about 15x) with only a slight accuracy cost compared with real objects.
Limitations
- During data collection only simplified geometric primitives are rendered, with complex objects substituted afterwards.
- Rearrangement generation performance lags ground truth, with predicted placement motions less stable than grasp motions.
Experimental setup
Datasets: FAVOR OmniObject3D
Benchmarks: FAVOR
Metrics: Contact Ratio Solid Intersection Volume Average Pairwise Distance PSKL-J Likert perceptual score
Baselines: FLEX INET w/o obj opt
Builds on: GOAL SAGA VoxPoser MDM OWL-ViT
Closest papers by meaning
- CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement (CVPR 2025 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.85)
- AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation (ICLR 2025 · similarity 0.85)
- Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model (CVPR 2025 · similarity 0.85)
- NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis (CVPR 2024 · similarity 0.85)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.