Research map / Video / Video generation
TransPixeler: Advancing Text-to-Video Generation with Transparency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends pretrained DiT text-to-video models (CogVideoX) to jointly generate RGB and alpha channels via appended alpha tokens, partial LoRA on alpha tokens, and attention rectification, trained on the 484-video VideoMatte240K dataset.
Problem
Generating RGBA (transparent) video from text is hard because RGBA video data is scarce (about 484 videos) and existing video models cannot produce alpha channels; prediction-based matting pipelines generalize poorly and suffer from weak RGB-alpha alignment.
Approach
Append alpha tokens to the DiT input sequence with reinitialized shared positional encodings and a learnable zero-initialized domain embedding, apply LoRA only to alpha-token projections, and use an attention mask that blocks Text-to-Alpha attention while keeping RGB-to-Alpha attention so RGB and alpha are jointly generated.
Key results
- Joint generation achieved 93.3% user preference for RGBA alignment vs 6.7% for AnimateDiff+LayerDiffusion in a 30-video, 87-user study.
- Quantitative evaluation (Fig. 11) reports a balance of low flow difference (alignment) and low FVD (preserved generation quality) against ablations; exact numbers are only in a figure.
- Ablations show removing RGB-to-Alpha attention causes alpha misalignment, and batch or latent-dimension extension alternatives yield weaker alignment or reduced diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TransPixeler + CogVideoX instead of AnimateDiff + LayerDiffusion | Text-to-RGBA video generation · User study (30 videos, 87 users) | RGBA alignment preference: 6.7% -> 93.3% | Helped |
| Use TransPixeler + CogVideoX instead of AnimateDiff + LayerDiffusion | Text-to-RGBA video generation · User study (30 videos, 87 users) | Motion quality preference: 21.7% -> 78.3% | Helped |
| Add RGB-attend-to-Alpha attention (ablation) | TransPixeler · RGBA video generation | RGB-alpha alignment | Helped |
| Add RGB-attend-to-Alpha attention (ablation) | TransPixeler · RGBA video generation | RGB motion quality | Helped |
| Allow Text-attend-to-Alpha attention (ablation) | TransPixeler · RGBA video generation | RGB generation quality | Hurt |
| Use batch extension instead of sequence extension (ablation) | TransPixeler · RGBA video generation | RGB-alpha alignment | Hurt |
| Use latent-dimension extension instead of sequence extension (ablation) | TransPixeler · RGBA video generation | generation diversity | Hurt |
Surprising findings
- Text-to-Alpha attention degrades RGB generation quality due to the domain gap between alpha (contour-only) and RGB, so it is blocked.
Limitations
- Training data is limited to 484 videos, mainly human subjects in VideoMatte240K, constraining diversity.
- Quantitative evaluation is small (80 videos) and relies on proxy metrics (optical flow difference, FVD) without ground-truth alpha.
- Matting baselines and the Lotus/SAM-2 comparisons are primarily qualitative.
Experimental setup
Datasets: VideoMatte240K ShareGPT4V captions
Benchmarks: FVD (vs original CogVideoX RGB outputs) optical flow difference (RGB-alpha alignment) user study on Amazon Mechanical Turk
Metrics: Flow Difference FVD user preference %
Baselines: Lotus + RGBA SAM-2 AnimateDiff + LayerDiffusion
Builds on: CogVideoX LoRA flow matching LayerDiffusion Lotus
Closest papers by meaning
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.87)
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (ICLR 2025 · similarity 0.86)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
- GenTron: Diffusion Transformers for Image and Video Generation (CVPR 2024 · similarity 0.86)
- SimDA: Simple Diffusion Adapter for Efficient Video Generation (CVPR 2024 · similarity 0.85)
- PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.