Research map / Video / Video segmentation and tracking
Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of video dataset distillation with a temporal-compression taxonomy, and a two-stage static-dynamic disentangled method that distills a static frame memory then learns a dynamic memory and integrator network.
Problem
Dataset distillation for videos is underexplored; naive frame-wise matching is costly and temporal information is poorly learned and storage-heavy.
Approach
Disentangle video distillation into a static memory (single-frame image distillation) and a learnable dynamic memory combined by an integrator network H, with a taxonomy of temporal compression (frame counts, segments, interpolation) guiding design.
Key results
- On MiniUCF IPC=1, FRePo+Ours reaches 22.0% vs 20.3% for FRePo while using about 42% of its storage; MTT+Ours reaches 23.3% vs 19.0% for MTT.
- Distilling a video into a single still image already yields over 17% accuracy on MiniUCF, and larger Nreal/Nsyn brings under 3% gain.
- On Kinetics-400 the method underperforms MTT at IPC=5 (7.0 vs 11.5 top-5 for MTT+Ours vs MTT per table, MTT baseline 9.1), attributed to shared H network across 400 classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add static-dynamic disentanglement to MTT | MTT on MiniUCF · MiniUCF, IPC 1 | top-1 accuracy (%): 19.0 ± 0.1 -> 23.3 ± 0.6 | Helped |
| Add static-dynamic disentanglement to FRePo | FRePo on MiniUCF · MiniUCF, IPC 5 | top-1 accuracy (%): 30.2 ± 1.7 -> 31.2 ± 0.7 | Helped |
| Add static-dynamic disentanglement to FRePo | FRePo on HMDB51 · HMDB51, IPC 1 | top-1 accuracy (%): 7.2 ± 0.8 -> 8.6 ± 0.5 | Helped |
| Add static-dynamic disentanglement to MTT | MTT on Kinetics-400 · Kinetics-400, IPC 5 | top-5 accuracy (%): 9.1 ± 0.3 -> 11.5 ± 0.5 | Helped |
| Add static-dynamic disentanglement to MTT | MTT on SSv2 · SSv2, IPC 1 | top-5 accuracy (%): 3.9 ± 0.1 -> 5.5 ± 0.1 | Helped |
| Use a parametric interpolator instead of duplication (ablation) | DM video distillation · MiniUCF, Nsyn=Nreal=2 | accuracy (%): 18.0 ± 0.4 -> 18.6 ± 0.8 | Helped |
| Use a parametric interpolator instead of linear interpolation (ablation) | DM video distillation · MiniUCF, Nsyn=Nreal=2 | accuracy (%): 15.6 ± 0.1 -> 18.6 ± 0.8 | Helped |
| Increase dynamic memory per class from 0 to 1 (ablation) | Static-dynamic method · MiniUCF, IPC 1, 1 static memory per class | top-1 accuracy (%): 13.7 ± 1.1 -> 17.5 ± 0.5 | Helped |
| Increase dynamic memory per class from 1 to 2 (ablation) | Static-dynamic method · MiniUCF, IPC 1, 1 static memory per class | top-1 accuracy (%): 17.5 ± 0.5 -> 19.6 ± 1.2 | Helped |
| Increase dynamic memory per class from 0 to 1 (ablation) | Static-dynamic method · MiniUCF, IPC 1, 2 static memories per class | top-1 accuracy (%): 20.4 ± 0.5 -> 22.3 ± 0.0 | Helped |
| Increase static memory per class from 1 to 2 with no dynamic memory (ablation) | Static-dynamic method · MiniUCF, IPC 1 | top-1 accuracy (%): 13.7 ± 1.1 -> 20.4 ± 0.5 | Helped |
| Use static-dynamic disentanglement rather than static-only distillation (ablation) | MiniUCF distilled data · MiniUCF, IPC 1 | top-1 accuracy (%): 13.7 ± 1.1 -> 17.5 ± 0.1 | Helped |
Surprising findings
- Distilling a video into a single still image retains over 17% accuracy, and increasing frame numbers yields only marginal gains at much higher cost.
- Static-DC (frame-duplicated static images) performs worst, but the static-dynamic method outperforms naive video distillation with less storage.
Limitations
- Gains on Kinetics-400 are weaker, attributed to a single shared H network across many classes.
- Static memory is trained with early stopping and the approach relies on selecting frames per epoch; dense temporal correspondence is deemed non-critical, which may not generalize.
- Evaluation of FRePo-distilled data is only a reference due to label learning.
Experimental setup
Datasets: UCF101 HMDB51 Kinetics-400 Something-Something V2 MiniUCF
Benchmarks: MiniUCF HMDB51 Kinetics-400 SSv2
Metrics: top-1 accuracy top-5 accuracy storage (MB/GB) training time GPU memory
Baselines: Random coreset Herding K-Center DM MTT FRePo Static-DC
Builds on: DC DM MTT FRePo
Closest papers by meaning
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.89)
- DFDNet: Disentangling and Filtering Dynamics for Enhanced Video Prediction (AAAI 2025 · similarity 0.87)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.86)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.86)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.85)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.