Research map / Video / Video generation
TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TalkCuts, a 164k-clip, 500+ hour 1080P dataset of multi-shot human speech videos with 2D/3D pose and text annotations, plus the Orator LLM-directed baseline for multi-shot speech video generation.
Problem
Existing human video datasets are short, single-shot, and lack multi-shot camera labels, 3D motion, and scale, so long-form multi-shot speech video generation cannot be studied well.
Approach
Curate a large YouTube speech-video dataset with per-clip shot-type labels, DWPose 2D keypoints, SMPL-X 3D motion, and VLM captions, and use an LLM "director" (RAG few-shot GPT-4o for shot planning) to orchestrate shot, gesture, and vocal instructions feeding a CosyVoice + CogVideoX/Hallo3-based video generator.
Key results
- RAG-based GPT-4o shot planning achieves 70.66% accuracy, 64.06% SMA and 48.10% IoU, the best SMA and IoU among compared methods.
- Orator (fine-tuned on TalkCuts) reaches FID 45.86 and FVD 622.76 versus 57.28 and 855.84 for the strongest baseline Hallo3 in audio-driven generation.
- Fine-tuning Animate Anyone on TalkCuts lowers FVD from 789.36 to 456.84 and ArcFace distance from 0.450 to 0.344 in pose-guided generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Animate Anyone on TalkCuts | Animate Anyone · Pose-guided speech video (TalkCuts) | FVD: 789.360 -> 456.842 | Helped |
| Fine-tune ControlNeXt on TalkCuts | ControlNeXt · Pose-guided speech video (TalkCuts) | FVD: 485.118 -> 480.210 | Helped |
| Fine-tune MusePose on TalkCuts | MusePose · Pose-guided speech video (TalkCuts) | FVD: 823.020 -> 1014.342 | Hurt |
| Fine-tune MusePose on TalkCuts | MusePose · Pose-guided speech video (TalkCuts) | FID: 106.760 -> 87.000 | Helped |
| Fine-tune on TalkCuts (two-stage Hallo3 training with multi-shot data) | Hallo3 · Audio-driven speech video (TalkCuts test) | FID: 57.28 -> 45.86 | Helped |
| Fine-tune on TalkCuts (two-stage Hallo3 training with multi-shot data) | Hallo3 · Audio-driven speech video (TalkCuts test) | Sync-C: 3.84 -> 4.35 (improves here); SadTalker 4.57 remains higher | Hurt |
| Use RAG-retrieved few-shot examples for GPT-4o shot planning instead of zero-shot (ablation) | GPT-4o · LLM-guided camera shot transitions | SMA: 48.34% -> 64.06% | Helped |
| Use RAG-retrieved few-shot examples for GPT-4o shot planning instead of zero-shot (ablation) | GPT-4o · LLM-guided camera shot transitions | IOU: 40.59% -> 48.10% | Helped |
| Use RAG retrieval instead of random few-shot examples for GPT-4o (ablation) | GPT-4o · LLM-guided camera shot transitions | Accuracy: 67.50% -> 70.66% | Helped |
| LoRA fine-tuning of Llama 3.1 8B on shot division corpus (ablation) | Llama 3.1 8B · LLM-guided camera shot transitions | Accuracy: 21.65% (RAG) -> 79.09% (Tune) | Helped |
| LoRA fine-tuning of Llama 3.1 8B on shot division corpus (ablation) | Llama 3.1 8B · LLM-guided camera shot transitions | IOU: Tune 30.06% vs Embedding Model 35.60% | Hurt |
Surprising findings
- Fine-tuned Llama 3.1 8B reaches 79.09% accuracy but lower IoU than the plain embedding model, indicating accuracy alone is misleading for shot planning.
- MusePose fine-tuning on TalkCuts increased FVD (823.02 to 1014.34), a counter-trend result not discussed in depth.
Limitations
- Orator is a simple baseline; multi-shot consistency still struggles with hand details and object interactions and half/full-body or side views degrade.
- Pose-guided methods still show temporal inconsistency and identity drift after fine-tuning.
- Dataset is sourced from YouTube speech content, with diversity limited to speech scenarios.
Experimental setup
Datasets: TalkCuts TikTok TED Talks UBC-Fashion Speech2Gesture UBody TalkSHOW BEAT2 Shot Division Corpus
Benchmarks: TalkCuts test set (50 clips) TalkCuts shot planning split
Metrics: Accuracy SMA IoU FID FVD PSNR SSIM LPIPS Sync-C Sync-D
Baselines: SadTalker EchoMimicV2 Hallo3 MagicAnimate MimicMotion Animate Anyone MusePose ControlNeXt
Builds on: CogVideoX Hallo3 CosyVoice Animate Anyone DWPose
Closest papers by meaning
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · similarity 0.90)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.88)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.88)
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking (NeurIPS 2025 · similarity 0.88)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.87)
- HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.