Research map / Video / Video-language understanding
ConViS-Bench: Estimating Video Similarity Through Semantic Concepts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces concept-based video similarity estimation (ConViS) and the ConViS-Bench benchmark of 610 human-annotated video pairs scored on five semantic concepts, then benchmarks LMMs, global embedding models, and concept-conditioned retrieval.
Problem
Global video similarity scores are hard to interpret and ignore which semantic aspects (action, subjects, objects, location, order) drive similarity; existing language-based video differencing benchmarks are narrow in concept and domain.
Approach
Define similarity as a concept-conditioned score s(V1,V2|C) over natural-language concepts, composable with weights, and evaluate LMMs and global representations by correlation with human Likert ratings on FineVideo-derived pairs spanning 16 domains.
Key results
- LLaVA-OV-7B achieves the highest LMM correlations with humans, e.g. 58.64 Spearman rho on main objects and 58.94 on location, while order of actions is hardest (best 41.25 for LLaVA-Video-7B).
- Among global representation methods, VQAScore with LLaVA-OV reaches the highest average correlation, and DINOv2 gets 57.4 rho on location.
- Concept-conditioned retrieval with LLaVA-OV-7B reaches F1@1 of 61.9 on main action and 69.3 on location, above random chance (27.8 to 31.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide fewer input frames per video (16 to 1) (ablation) | LLaVA-OV-7B · ConViS-Bench — Main Action | Spearman ρ ×100: 51.76 -> 32.36 | Hurt |
| Provide fewer input frames per video (16 to 1) (ablation) | Qwen2.5-VL-7B · ConViS-Bench — Main Action | Spearman ρ ×100: 37.88 -> 8.31 | Hurt |
| Provide fewer input frames per video (16 to 1) (ablation) | InternVL2.5-4B · ConViS-Bench — Main Action | Spearman ρ ×100: 13.23 -> 28.11 | Helped |
| Provide fewer input frames per video (16 to 1) (ablation) | InternVL3-8B · ConViS-Bench — Main Action | Spearman ρ ×100: 40.69 -> 40.10 | No clear effect |
| Use concept-conditioned video similarity scores for retrieval rather than random choice | LMMs · ConViS-Bench — concept-conditioned retrieval | F1@1: Random Choice: 27.8; 30.4; 31.5; 28.4. Model averages: 52.6; 58.9; 54.6; 61.1 | Helped |
| Use more frames per video (ablation) | LLaVA-OV, LLaVA-Video, and Qwen2.5-VL · ConViS-Bench — Main Action | Spearman ρ ×100: +29.57/19.40/13.56% | Helped |
Surprising findings
- InternVL models, which may have seen the FineVideo evaluation videos in pre-training, struggle to align with human judgments and show flat performance as frame count drops, suggesting memorization rather than temporal reasoning.
- Concept correlations with humans are generally low (Krippendorff alpha 0.244 to 0.361), indicating noisy human judgments.
Limitations
- Concept set is limited to five general-purpose concepts and may miss domain-specific aspects of similarity.
- The benchmark is modest in size (610 pairs) and human inter-annotator agreement is low.
- Some models may have been pre-trained on FineVideo, confounding evaluation.
Experimental setup
Datasets: FineVideo ConViS-Bench
Benchmarks: ConViS-Bench
Metrics: Spearman rho Kendall tau R@1 P@1 F1@1
Baselines: VideoMAE DINOv2 CLIP InternVideo Gemini-2.0-Flash VQAScore CLIPScore
Builds on: FineVideo VidDiffBench StepDiff DINOv2 Sentence-BERT
Closest papers by meaning
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.87)
- Unbiasing through Textual Descriptions: Mitigating Representation Bias in Video Benchmarks (CVPR 2025 · similarity 0.87)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.87)
- Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding (CVPR 2024 · similarity 0.86)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.86)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.