AI Paper Atlas

Research map / Video / Video-language understanding

Aligning Step-by-Step Instructional Diagrams to Video Demonstrations

Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, Stephen Gould

CVPR 2023 · Video › Video-language understanding · approach: Video-language pretraining · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use the proposed three-loss contrastive framework instead of CLIPvideo-diagram alignment · IAW step diagramsTop1 Acc.%: 19.61 -> 28.62Improved
use the proposed framework instead of CLIPvideo-diagram alignment · IAW step diagramsR@1: 16.94 -> 22.30Improved
add sinusoidal progress rate features (SPRF) (ablation)Ours · IAW step diagramsTop1 Acc.%: 21.73 -> 28.62Improved
add sinusoidal progress rate features (SPRF) (ablation)Ours · IAW step diagramsAIE: 6.018 -> 3.734Improved
apply DTW instead of no post-processing (ablation)Ours · IAW step diagramsR@3: 45.00 -> 32.45Hurt
apply OT set matching (ablation)Ours · IAW step diagramsR@1: 22.30 -> 26.62Improved
use Video-Diagram loss A instead of standard CLIP infoNCE (ablation)contrastive alignment model · IAW step diagramsTop1 Acc.%: 19.61 -> 20.87Improved
use Video-Manual loss B (ablation)contrastive alignment model · IAW step diagramsTop1 Acc.%: 19.61 -> 27.20Improved
combine Video-Manual loss B with Video-Diagram loss A (ablation)contrastive alignment model · IAW step diagramsTop1 Acc.%: 27.20 -> 28.20Improved
add Intra-Manual loss C to losses A and B (ablation)contrastive alignment model · IAW step diagramsTop1 Acc.%: 28.20 -> 28.43Mixed
include page-diagram losses in addition to step-diagram losses (ablation)loss configuration D1 · IAW step diagramsTop1 Acc.%: 28.62 -> 28.26No clear effect
use page diagrams as training supervision in addition to step diagrams (ablation)loss configuration D1 · IAW page diagramsTop1 Acc.%: 33.72 -> 34.94Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy