Research map / Video / Video generation
InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstanceCap, an instance-aware structured video captioning pipeline (detection/segmentation with blurred backgrounds, then CoT MLLM annotation), releases a 22K InstanceVid dataset, and an inference-time InstanceEnhancer prompt expander for text-to-video training and generation.
Problem
Video captions used for text-to-video training suffer from insufficient detail, hallucinations, and imprecise motion description, which reduces the fidelity and instruction-following of generated videos.
Approach
Decompose each video into instances using an auxiliary model cluster (object detection, SAM2 segmentation, camera-motion detection), blur non-instance regions so the MLLM annotates each instance in isolation with class-specific hints, and refine dense captions into structured per-instance phrases via an improved chain-of-thought process.
Key results
- InstanceCap reaches 143.00 3DVAE score and 0.916 VQAScore on reconstruction-via-recaption, the best of all captioners compared (MiraData 144.75 / 0.905).
- On Inseval, Open-Sora fine-tuned with InstanceCap captions reaches 45.00% average accuracy vs 28.63% for base Open-Sora and 43.00% for CogVideoX-5B.
- Human user study scores InstanceCap at 4.60 on instance detail vs 3.35 for MiraData, and hallucination score 4.12 vs 3.21 (higher is better).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Open-Sora on InstanceVid using InstanceCap captions | Open-Sora v1.2-1.1B · Inseval | Average: 28.63% -> 45.00% | Helped |
| Use InstanceCap captions rather than MiraData captions for fine-tuning | Open-Sora v1.2-1.1B · Inseval | Average: 30.63% -> 45.00% | Helped |
| Remove camera movement hints (ablation) | InstanceCap · OpenVid-1M reconstruction-via-recaption | 3DVAEscore: 143.00 -> 144.38 | Hurt |
| Remove human-designed class hints (ablation) | InstanceCap · OpenVid-1M reconstruction-via-recaption | 3DVAEscore: 143.00 -> 143.87 | Hurt |
| Replace instance visual prompting with a red circle (ablation) | InstanceCap · OpenVid-1M reconstruction-via-recaption | 3DVAEscore: 143.00 -> 144.12 | Hurt |
| Replace blurred background masking with a blue background (ablation) | InstanceCap · OpenVid-1M reconstruction-via-recaption | 3DVAEscore: 143.00 -> 144.38 | Hurt |
| Remove InstanceEnhancer at inference (ablation) | Open-Sora v1.2-1.1B fine-tuned on InstanceVid · Inseval | Average: 37.88% -> 45.00% | Hurt |
| Fine-tune Open-Sora using InstanceCap captions | Open-Sora v1.2-1.1B · EvalCrafter | Flow Score: 22.84 -> 23.19 | Helped |
| Fine-tune Open-Sora using InstanceCap captions | Open-Sora v1.2-1.1B · EvalCrafter | CLIP Score: 68.19 -> 68.88 | Helped |
Surprising findings
- Using a red-screen background to isolate instances hurt MLLM captioning, while blurring non-instance regions worked better.
Limitations
- Precision depends on object detection, which needs fine-tuning for domain-specific instances.
- Benefits diminish in instance-free scenes.
- InstanceVid scale (22K) limits its use as a large-scale pretraining dataset.
Experimental setup
Datasets: OpenVid-1M InstanceVid EvalCrafter Inseval
Benchmarks: Inseval EvalCrafter VQAScore reconstruction benchmark (500 OpenVid-1M clips)
Metrics: 3DVAEscore VQAScore human evaluation instance detail and hallucination scores Inseval accuracy by action/color/shape/texture/detail EvalCrafter action, motion, flow, CLIP, BLIP-BLEU, SD scores
Baselines: Panda-70M ShareGPT4Video MiraData LLaVA-Video-72B CogVideoX-5B Pyramid-Flow Open-Sora Plan
Builds on: LLaVA-Video SAM2 OpenVid-1M Open-Sora CogVideoX
Closest papers by meaning
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.88)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.87)
- Can Text-to-Video Generation help Video-Language Alignment? (CVPR 2025 · similarity 0.86)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.