AI Paper Atlas

Research map / Video / Video-language understanding

Free-VSC: Free Semantics from Visual Foundation Models for Unsupervised Video Semantic Compression

Yuan Tian, Guo Lu, Guangtao Zhai

ECCV 2024 · Video › Video-language understanding · approach: Long video understanding · 18 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace VFM semantic objective with single MAE SSL objective (60K videos) (ablation)Free-VSC · UCF101 @0.02bpp (TSM)Top1 accuracy: over 5% accuracy dropHurt
Add a second SSL objective and use all 230K videos instead of 1 SSL on 60K (ablation)Free-VSC with SSL objective · UCF101 @0.02bpp (TSM)Top1 accuracy: about +2% (still about 2% below VFM model)Improved
Add a second VFM (DINOv2 with VideoMAEv2) (ablation)Free-VSC · UCF101 @0.02bpp (TSM)Top1 accuracy: 83.11 -> 84.11Improved
Add third VFM SSL-Swin (ablation)Free-VSC · UCF101 @0.02bpp (TSM)Top1 accuracy: 84.11 -> 85.07Improved
Add VFM-specific prompts to the shared alignment transformer (ablation)Prom-SAL · UCF101 @0.02bpp (TSM)Top1 accuracy: 80.92 -> 85.07Improved
Share Prom-SAL transformer parameters across VFMs (ablation)Prom-SAL · UCF101 @0.02bpp (TSM)Top1 accuracy: 83.01 -> 85.07Improved
Use vanilla Transformer rather than 3D CNN for the alignment layer (ablation)Prom-SAL · UCF101 @0.02bpp (TSM)Top1 accuracy: 79.56 -> 85.07Improved
Use both coarse and fine VFM features instead of one level (ablation)Free-VSC · UCF101 @0.02bpp (TSM)Top1 accuracy: 84.28 -> 85.07 (vs fine only)Improved
Use video-adaptive dynamic and region-wise trajectory over static trajectory (ablation)entropy model · UCF101 @0.02bpp (TSM)BD-Rate: bitcost +21.7% without both designsImproved
Use trajectory-based entropy model instead of motion compensation or ConvLSTM (ablation)inter-frame entropy model · UCF101 @0.02bpp (TSM)Top1 accuracy: about 1.5% and 2.2% accuracy drops for motion and ConvLSTMImproved
Increase number of trajectories S from 5 to 10 (ablation)entropy model · UCF101 @0.02bpp (TSM)BD-Rate: 0% -> -1.8%No clear effect
Apply VFM semantic learning to DCVC-DC and SMC (ablation)DCVC-DC and SMC · UCF101 @0.02bpp (TSM)Top1 accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy