Research map / Video / Video generation
HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HOIGen-1M, a million-clip human-object interaction video dataset with MoME-based hallucination-checked captions and two HOI metrics (CoarseHOIScore, FineHOIScore) for evaluating T2V models.
Problem
Current text-to-video models struggle to generate human-object interactions because no large-scale HOI videos with accurate captions exist, and there are no metrics specific to HOI generation quality.
Approach
Curate over one million high-quality HOI videos via metadata/aesthetic/motion filtering plus MLLM and human verification, caption them with a Mixture-of-Multimodal-Experts strategy that cross-verifies multiple MLLMs to detect and correct hallucinations, and score generated videos with an HOI detector-based coarse metric and a contact-based fine metric.
Key results
- Kling 1.5 reaches only 42.72% CoarseHOIScore, the best of ten evaluated T2V models, while Gen-3 scores 23.98% on the fine-grained metric's counterpart column; open models score lower.
- Fine-tuning CogVideoX-5B on HOIGen-1M raises CoarseHOIScore from 32.84% to 44.04%, and CogVideoX-2B from 31.34% to 39.13%.
- Captions from the MoME strategy raise CoarseHOIScore from 33.17% (PLLaVA captions) to 39.13% when training CogVideoX-2B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed MoME captions instead of PLLaVA captions (ablation) | T2V model fine-tuned on HOIGen-1M · HOI video generation | CoarseHOIScore: 33.17% -> 39.13% | Helped |
| use the proposed MoME captions instead of PLLaVA captions (ablation) | T2V model fine-tuned on HOIGen-1M · HOI video generation | FineHOIScore: 92.78% -> 93.82% | Helped |
| fine-tune on HOIGen-1M (ablation) | OpenSora · HOI video generation | CoarseHOIScore: 31.86% -> 35.38% | Helped |
| fine-tune on HOIGen-1M (ablation) | OpenSora · HOI video generation | FineHOIScore: 91.86% -> 94.91% | Helped |
| fine-tune on HOIGen-1M (ablation) | CogVideoX-2B · HOI video generation | CoarseHOIScore: 31.34% -> 39.13% | Helped |
| fine-tune on HOIGen-1M (ablation) | CogVideoX-2B · HOI video generation | FineHOIScore: 92.06% -> 93.82% | Helped |
| fine-tune on HOIGen-1M (ablation) | CogVideoX-5B · HOI video generation | CoarseHOIScore: 32.84% -> 44.04% | Helped |
| fine-tune on HOIGen-1M (ablation) | CogVideoX-5B · HOI video generation | FineHOIScore: 94.25% -> 96.04% | Helped |
Surprising findings
- VBench motion smoothness exceeds 98% for nearly all models even though generated interactions are clearly unreasonable, so VBench does not capture HOI quality.
- Out-of-distribution prompts score nearly the same as in-distribution prompts (63.20% vs 61.20%), suggesting possible emergent interaction generation.
Limitations
- Million-scale data may still be insufficient as T2V model sizes grow.
- CoarseHOIScore and FineHOIScore have a remaining gap with human preferences.
- Human verification of 80M raw clips took about eight weeks and relies on seven annotators.
Experimental setup
Datasets: BEHAVE InterCap HOI4D Panda-70M ViSR Mixkit Pixabay Pexels HICO-DET
Benchmarks: VBench HOIGen-1M prompt suite (306 prompts)
Metrics: CoarseHOIScore FineHOIScore subject consistency background consistency motion smoothness aesthetic quality imaging quality temporal flickering overall consistency
Baselines: OpenSora OpenSoraPlan CogVideoX-2B CogVideoX-5B Mochi-10B Gen-3 Pika Dreamina Hailuo Kling 1.5
Builds on: VBench PLLaVA CogVideoX OpenSoraPlan
Closest papers by meaning
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- Semantic-Aware Human Object Interaction Image Generation (ICML 2024 · similarity 0.89)
- Auto-Regressive Diffusion for Generating 3D Human-Object Interactions (AAAI 2025 · similarity 0.88)
- OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation (CVPR 2025 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.