Research map / Video / Video segmentation and tracking
VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VEDiT, a diffusion-transformer (flow matching) predictor that forecasts unseen clip embeddings from frozen SigLIP features for procedural video tasks, trained without pretraining or language supervision.
Problem
Procedural video representation learning (step forecasting, task classification, procedure planning, long-horizon action anticipation) typically relies on expensive large-scale pretraining of the prediction model with noisy language/ASR supervision; the necessity of this is unclear.
Approach
Use a frozen pretrained visual encoder's clip-level embeddings and a DiT-style flow-matching denoiser with joint attention between seen and target clip embeddings and RoPE temporal encoding, trained only with downstream cross-entropy and no pretraining or text supervision.
Key results
- On COIN step forecasting, SigLIP+VEDiT reaches 51.8% top-1 vs. 46.8% for ProceduralVRL; task classification 94.6% vs. 90.8%.
- Replacing SCHEMA's blocks with VEDiT improves NIV SR by up to 2.28% and COIN SR by up to 5.89% at T=3.
- On Ego4D-v2 long-term anticipation, VEDiT lowers Verb ED@20 to 0.697 (from 0.718) and Noun ED@20 to 0.711 (from 0.742) versus the SigLIP-based baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the SigLIP + VEDiT prediction setup with the proposed model on COIN step forecasting | COIN step forecasting · COIN | accuracy: 46.8 -> 51.8 | Helped |
| replace the SigLIP + VEDiT prediction setup with the proposed model on COIN task classification | COIN task classification · COIN | accuracy: 90.8 -> 94.6 | Helped |
| replace the Ego4D baseline transformer with VEDiT using SigLIP | Ego4D-v2 long-horizon action anticipation · Ego4D-v2 validation | Verb ED@20: 0.718 -> 0.697 | Helped |
| replace the Ego4D baseline transformer with VEDiT using SigLIP | Ego4D-v2 long-horizon action anticipation · Ego4D-v2 validation | Noun ED@20: 0.742 -> 0.711 | Helped |
| replace SCHEMA decoder blocks with VEDiT blocks | SCHEMA procedure planning · NIV, COIN, and CrossTask; T=3 and T=4 | SR, mAcc, and mIoU: NIV gains 0.97%-2.28% SR, 1.92%-3.39% mAcc, 0.49%-0.90% mIoU; COIN gains 2.21%-5.89% SR, 2.31%-7.07% mAcc, 0.86%-2.57% m | Mixed |
| use SigLIP instead of SSL visual encoders (ablation) | VEDiT on COIN · COIN | step forecasting and task classification accuracy: 47.03 -> 50.05 step forecasting; 90.89 -> 94.38 task classification | Helped |
| add HowTo100M pretraining with HowToCaption temporal information (ablation) | SigLIP + VEDiT on COIN · COIN | step forecasting accuracy: 51.8 -> 52.1 | Helped |
| add HowTo100M pretraining with HowToCaption temporal information (ablation) | SigLIP + VEDiT on COIN · COIN | task classification accuracy: 94.6 -> 94.6 | No clear effect |
| use joint attention instead of self-attention (ablation) | VEDiT · COIN | step forecasting accuracy: 49.7 -> 50.3 | Helped |
| use joint attention instead of self-attention (ablation) | VEDiT · COIN | task classification accuracy: 94.3 -> 94.4 | Helped |
| use 36 denoising steps instead of single-step denoising (ablation) | VEDiT · COIN step forecasting | top-1 accuracy: 50.77 -> 51.01 | Helped |
| vary denoising steps to 4 or greater than 44 (ablation) | VEDiT · COIN step forecasting | optimization | Hurt |
Surprising findings
- Large-scale HowTo100M pretraining of the predictor gives only a marginal 0.3% gain on COIN step forecasting (52.1 vs 51.8).
Limitations
- Gains on procedure planning are small (often under 1%), and CrossTask shows a slight drop on mIoU at T=3.
- Pretraining objective explored is limited; pretraining benefits may depend on the objective and data quality.
- Code is promised only after peer review; the authors have not released it.
Experimental setup
Datasets: COIN CrossTask NIV Ego4D-v2 HowTo100M HowToCaption
Benchmarks: COIN CrossTask NIV Ego4D-v2
Metrics: top-1 accuracy success rate mAcc mIoU edit distance ED@20
Baselines: DistantSup ProceduralVRL SCHEMA DDN P3IV EGPP Ext-GAIL PaMsEgoAI Ego4D Baseline TimeSformer VideoCLIP
Builds on: DiT rectified flow / flow matching SCHEMA ProceduralVRL RoPE
Closest papers by meaning
- Seer: Language Instructed Video Prediction with Latent Diffusion Models (ICLR 2024 · similarity 0.87)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.87)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.86)
- Separating Knowledge and Perception with Procedural Data (ICML 2025 · similarity 0.86)
- VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting (AAAI 2025 · similarity 0.86)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.