Research map / Video / Video generation
BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes blob video representations (tilted-ellipse parameters per frame plus sparse free-form blob captions) and BlobGEN-Vid, a blob-grounded video diffusion framework with masked 3D self-attention and context interpolation, built on U-Net (VideoCrafter2) and DiT (CogVideoX-5B) backbones.
Problem
Text-to-video models struggle with complex multi-object prompts and offer limited controllability over object layout, motion, appearance transitions, and temporal consistency.
Approach
Represent each object per frame as a tilted ellipse (blob parameters) with sparse free-form captions, fuse them via masked spatial cross-attention and a masked 3D self-attention that restricts each object to attend to itself across frames, and linearly interpolate caption embeddings between anchor frames; the framework is added to frozen pretrained video diffusion models.
Key results
- On YoutubeVIS-2021, BlobGEN-Vid (VC2) reaches mIOU 0.6119 vs 0.4916 for TrackDiffusion (over 20% gain), and CogVideoX-5B variant reaches rCLIPt 0.2888 and rCLIPi 0.8364.
- Adding masked 3D self-attention improves FVD by 40% and mIOU by 117% over the linear-interpolation baseline without it (Table 4, rows 1-2).
- GPT-4o + BlobGEN-Vid outperforms proprietary generators such as Gen-3 Alpha and Kling 1.0 on several T2V-CompBench compositional dimensions (e.g., Numeracy 0.3910 vs 0.3044 for Kling 1.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use blob-grounded generation instead of TrackDiffusion | BlobGEN-Vid (CogVideoX-5B) · YoutubeVIS-2021 | mIOU: 0.4916 -> 0.5982 | Helped |
| Use GPT-4o blob planning with BlobGEN-Vid instead of the strongest listed baseline | T2V-CompBench · T2V-CompBench | Spatial: 0.6150 -> 0.6725 | Helped |
| Use GPT-4o blob planning with BlobGEN-Vid instead of the strongest listed baseline | T2V-CompBench · T2V-CompBench | Consist-Attr: 0.8045 -> 0.7400 | Hurt |
| Add masked 3D self-attention (ablation) | BlobGEN-Vid on ScanNet++ · ScanNet++ test | PSNR: 21.71 -> 22.20 | Helped |
| Add masked 3D self-attention (ablation) | BlobGEN-Vid on ScanNet++ · ScanNet++ test | CFC: 0.9432 -> 0.9453 | Helped |
| Add masked 3D self-attention (ablation) | BlobGEN-Vid on YoutubeVIS-2021 · YoutubeVIS-2021 | mIOU: 0.2585 -> 0.5623 | Helped |
| Replace linear interpolation with Slerp (ablation) | BlobGEN-Vid · YoutubeVIS-2021 | mIOU: 0.5771 -> 0.5702 | Hurt |
| Replace linear interpolation with Perceiver (ablation) | BlobGEN-Vid · YoutubeVIS-2021 | mIOU: 0.5771 -> 0.5926 | Helped |
| Increase training data from 160K to 400K videos (ablation) | BlobGEN-Vid with linear interpolation · YoutubeVIS-2021 | mIOU: 0.5623 -> 0.5771 | Helped |
| Increase training data from 400K to 1M videos (ablation) | BlobGEN-Vid with Perceiver interpolation · YoutubeVIS-2021 | mIOU: 0.5926 -> 0.6119 | Helped |
Surprising findings
- Spherical interpolation (slerp) of caption embeddings did not help over linear interpolation, which the authors attribute to CLIP text embedding topology.
- FVD may not be a robust metric for this setting, and higher rCFC does not necessarily indicate better performance.
Limitations
- FVD is not a robust metric for evaluating layout-guided video quality
- Blob captions only provided every k frames, relying on interpolation for non-anchor frames
- Training data annotation pipeline depends on multiple pretrained models (Grounding DINO/ODISE, SAM2, LLaVA-NeXT), which may propagate errors
Experimental setup
Datasets: OpenVid-1M VidGEN-1M HD-VILA ScanNet++ YoutubeVIS-2021
Benchmarks: YoutubeVIS-2021 T2V-CompBench TC-Bench ScanNet++
Metrics: FVD mIOU rCLIPt rCLIPi rCFC TCR TC-Score FID IS CLIP similarity
Baselines: TrackDiffusion LVD VideoTetris BlobGEN-3D Open-Sora v1.2 CogVideoX-5B Pika 1.0 Dream Machine Kling 1.0 Gen-3 Alpha
Builds on: BlobGEN GLIGEN VideoCrafter2 CogVideoX
Closest papers by meaning
- Compositional Text-to-Image Generation with Dense Blob Representations (ICML 2024 · similarity 0.92)
- Compositional 3D-aware Video Generation with LLM Director (NeurIPS 2024 · similarity 0.88)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.88)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.87)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.