Research map / Large language models / Agents, code and math
ChatHuman: Chatting about 3D Humans with Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a LLaVA-1.5-7B-based agent with LoRA to select and invoke 26 3D-human tools, using paper-based RAG for tool documentation and LLM-based discrimination and integration of tool outputs.
Problem
Non-expert users cannot easily select, apply, and interpret the many specialized academic tools for 3D human tasks (pose, shape, contact, generation), and general LLMs lack the domain knowledge to use them.
Approach
Equip a multimodal LLM agent with paper-based retrieval-augmented tool documentation, graph-structured tool invocation (node, chain, DAG), and tool-conditioned transformation of 3D outputs into text or images so the LLM can discriminate among and integrate tool results.
Key results
- On the tool-usage benchmark, ChatHuman reaches 0.970 execution success rate on seen tools and 0.954 on unseen tools versus 0.753 for Visual ChatGPT-4 on seen tools.
- On reasoning-based pose estimation (RPE), ChatHuman reduces MPJPE to 147.2 mm versus 225.2 mm for HMR 2.0 and 253.6 mm for ChatPose.
- On DECO HOI contact detection, ChatHuman reaches F1 0.63 versus 0.39 for LLaVA-1.5 without tools.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add paper-based RAG for unseen tools (ablation) | ChatHuman · Unseen tools | SR: 0.822 -> 0.891 | Helped |
| Add tool-paper content to training data alongside paper-based RAG (ablation) | ChatHuman · Unseen tools | SR: 0.891 -> 0.954 | Helped |
| Add paper-based RAG and tool-paper content to training (ablation) | ChatHuman · Seen tools | SR: 0.960 -> 0.970 | Helped |
| Use tools for human-object contact detection (ablation) | ChatHuman · DECO test set | F1 score: 0.39 -> 0.63 | Helped |
| Use tools for body-shape measurement (ablation) | ChatHuman · HBW | Chest error (cm): 16.5 -> 6.1 | Helped |
| Increase the tool combination from 3 to 5 tools (ablation) | ChatHuman · Tool combination evaluation | SR: 0.932 -> 0.847 | Hurt |
| Select between Tool A and Tool B pose estimates (ablation) | ChatHuman · MixPose | MPJPE (mm): 126.2 -> 119.6; 124.0 -> 119.6 | Helped |
| Modify tool body-shape measurements using agent judgment (ablation) | ChatHuman · HBW | Chest error (cm): 8.6 -> 6.1 | Helped |
| Use ChatHuman instead of ChatPose for reasoning-based pose estimation | RPE Benchmark · RPE Benchmark | MPJPE: 253.6 -> 147.2 | Helped |
| Use ChatHuman instead of PoseScript for classical text-to-pose generation | ChatHuman · PoseScript | RP2T top 5 recall: 40.4 -> 41.8 | Helped |
| Use ChatHuman instead of ChatPose for speculative pose generation | ChatHuman · SPG Benchmark | RP2T top 5 recall: 3.3 -> 3.2 | Hurt |
Surprising findings
- Adding tools yields little change on classical 3DPW pose estimation (ChatHuman 91.3 mm vs HMR 2.0 91.0 mm), so gains come mainly from reasoning-based tasks.
- Visual ChatGPT-4 zero-shot achieves strong tool-use accuracy, close to the fine-tuned ChatHuman on unseen tools.
Limitations
- Fails when user requests are vague and the LLM cannot correct an incorrect initial tool call.
- Performance is bounded by the accuracy of the underlying academic tools and training labels (e.g., height labels ignoring shoe height).
- Classical pose estimation accuracy is only comparable to, not better than, the specialist tools.
Experimental setup
Datasets: COCO 3DPW MOYO PoseScript SHAPY DECO HBW RPE SPG
Benchmarks: 3DPW RPE PoseScript SPG HBW DECO MixPose
Metrics: SR SRt SRact SRargs IoU MPJPE PA-MPJPE MPJRE recall@K anthropometric error
Baselines: GPT4Tools GPT4Tools-FT Visual ChatGPT ChatPose LLaVA-1.5 CLIFF-BEDLAM HMR 2.0 SPIN PoseScript
Builds on: LLaVA GPT4Tools Visual ChatGPT ChatPose LoRA
Closest papers by meaning
- ChatPose: Chatting about 3D Human Pose (CVPR 2024 · similarity 0.91)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.88)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)
- Generative Proxemics: A Prior for 3D Social Interaction from Images (CVPR 2024 · similarity 0.86)
- GenZI: Zero-Shot 3D Human-Scene Interaction Generation (CVPR 2024 · similarity 0.86)
- MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.