Research map / Video / Video generation
F³-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes F³-Pruning, a training-free pruning method that removes low-aggregate-attention temporal attention weights in CogVideo and Tune-A-Video to speed up text-to-video inference while improving FVD.
Problem
Inference of large text-to-video transformer and diffusion models is costly, and prior acceleration methods either need retraining or are specific to one model family.
Approach
Observes that temporal attention is redundant and decreasingly important over layers/denoising steps, and prunes the temporal attention weights in layers or timesteps whose aggregate attention score (sum of temporal attention values) ranks in the lowest fraction, with no retraining and no per-inference merging overhead.
Key results
- On UCF-101 with CogVideo, F³-Pruning reduces inference time from 486 s to 359 s (about 1.35x speedup) and improves FVD from 2537 to 1990 relative to unpruned CogVideo's reported baseline row.
- On LONGTEXT with Tune-A-Video, F³-Pruning lowers time from 45.08 s to 43.83 s and raises CLIP(text) from 32.83 to 33.30 and CLIP(img) from 97.78 to 97.95.
- Among pruning methods on CogVideo, F³-Pruning achieves the best FVD on UCF-101 (1990) and DAVIS (1781) and the fastest time (359 s), while ToMe and others increase inference time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply F³-Pruning to temporal attention | CogVideo · UCF-101 | FVD: 2537 -> 1990 | Helped |
| apply F³-Pruning to temporal attention | CogVideo · UCF-101 | Time (s): 486 -> 359 | Helped |
| apply F³-Pruning to temporal attention | CogVideo · UCF-101 | Clip(text): 25.03 -> 24.77 | Hurt |
| apply F³-Pruning to temporal attention | CogVideo · DAVIS | FVD: 1851 -> 1781 | Helped |
| apply F³-Pruning to temporal attention | CogVideo · LONGTEXT | Clip(text): 28.07 -> 28.4 | Helped |
| apply F³-Pruning | Tune-A-Video · LONGTEXT | Clip (text): 32.83 -> 33.30 | Helped |
| apply F³-Pruning | Tune-A-Video · LONGTEXT | Time (s): 45.08 -> 43.83 | Helped |
| remove CA pruning (prune CA instead of TA) (ablation) | CogVideo · UCF-101 | FVD: 5357 -> 4730 | Helped |
| remove SA pruning (prune SA instead of TA) (ablation) | CogVideo · UCF-101 | FVD: 5357 -> 5410 | Hurt |
| prune TA (ablation) | CogVideo · UCF-101 | FVD: 5357 -> 4239 | Helped |
| set pruning ratio α=25% (ablation) | CogVideo · UCF-101 | FVD: 5357 -> 4937 | Helped |
| set pruning ratio α=50% (ablation) | CogVideo · UCF-101 | FVD: 5357 -> 4239 | Helped |
Surprising findings
- Pruning temporal attention improves FVD and text alignment rather than only trading off quality for speed.
- Token-merging baselines (Pruning, Reorganization, TPS, ToMe) increased inference time despite reducing FLOPs.
Limitations
- Pruning ratio is fixed at 50% per the main experiments; higher ratios (75%) substantially hurt text-video alignment and coherence.
- Evaluated only on two T2V models and three datasets; FVD relies on a non-standard pretraining setup for the metric as described.
- The pruning ratio must be chosen empirically and the AAS calibration requires running inference on a public dataset first.
Experimental setup
Datasets: UCF-101 DAVIS LONGTEXT
Benchmarks: UCF-101 DAVIS LONGTEXT
Metrics: FVD CLIP score (text) CLIP score (image) PFLOPS/TFLOPS inference time (s)
Baselines: Token Pruning Token Reorganization TPS ToMe ToMeSD Tune-A-Video
Builds on: ToMe Tune-A-Video CogVideo
Closest papers by meaning
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.90)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.88)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.88)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.88)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.88)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.