Research map / Video / Video-language understanding
Identity-Text Video Corpus Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Identity-Text Video Corpus Grounding (ITVCG), builds the TVR-IT dataset with identity images from TVR TV shows, and proposes Video-Locator, a Vicuna-7B Video LLM with facial adapters for joint identity-text-video retrieval and moment localization.
Problem
Video corpus grounding uses only textual queries and cannot distinguish visual identities in videos, so retrieval of moments involving specific people is inaccurate.
Approach
Replace identity names in text queries with identity face images, and use a Video LLM with separate facial and visual adapters, splitting its layers into a video-identity-text alignment stage (contrastive retrieval) and a fine-grained fusion stage (moment localization).
Key results
- Video-Locator reaches 16.86 VCG IoU=0.5 on test-seen vs 12.30 for SQuiDNet (best baseline).
- On test-unseen, Video-Locator reaches 41.19 VR R@1 versus 35.45 for SQuiDNet, but its unseen-identity performance lags its seen-identity performance.
- Re-ranking by identity count raises test-unseen VCG IoU=0.5 from 4.64 to 6.27 without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add identity images and face features to the text query and video representation (ablation) | Video-Locator · TVR-IT test-seen | VCG IoU≥0.5: 7.12 -> 16.86 | Helped |
| Add identity images and face features to the text query and video representation (ablation) | Video-Locator · TVR-IT test-unseen | VCG IoU≥0.5: 6.29 -> 4.64 | Hurt |
| Pretrain the visual adapter on image-text pairs (ablation) | Video-Locator · TVR-IT test-seen | VCG IoU≥0.5: 14.29 -> 16.86 | Helped |
| Add contrastive video-query alignment loss Lcl (ablation) | Video-Locator · TVR-IT test-seen | VR R@1: 0.04 -> 36.04 | Helped |
| Add Ll1 localization loss (ablation) | Video-Locator · TVR-IT test-seen | VCG IoU≥0.5: 15.32 -> 16.86 | Helped |
| Add Lgiou localization loss (ablation) | Video-Locator · TVR-IT test-seen | VCG IoU≥0.5: 13.48 -> 16.86 | Helped |
| Add Lbce auxiliary frame-classification loss (ablation) | Video-Locator · TVR-IT test-unseen | VCG IoU≥0.5: 4.19 -> 4.64 | Helped |
| Rerank retrieved videos by the number of matching query identities | Video-Locator · TVR-IT test-unseen | VCG IoU≥0.5: 4.64 -> 6.27 | Helped |
| Rerank retrieved videos by the number of matching query identities | Video-Locator · TVR-IT test-seen | VCG IoU≥0.5: 16.86 -> 17.35 | Helped |
Surprising findings
- Video-Locator-T (no identity input) outperforms Video-Locator on the test-unseen split, indicating the model overfits to seen identities and fails to generalize to unseen ones.
Limitations
- Strong overfitting to the identities seen in training; unseen-identity performance is much lower.
- Identity set is small (463 identities from 6 TV shows).
- Face features are hard to describe in language, so pretraining with face data was difficult.
Experimental setup
Datasets: TVR TVR-IT LCS-558K
Benchmarks: TVR-IT
Metrics: R@1 R@10 R@100 IoU=0.5 IoU=0.7
Baselines: XML ReLoCLNet SQuiDNet Video-Locator-T
Builds on: TVR LLaVA InfoNCE LoRA
Closest papers by meaning
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.85)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.84)
- ARGUS: Stacked Multi-View Identity Mosaic Injection for Subject-Preserving Video Generation (NeurIPS 2026 · similarity 0.84)
- VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · similarity 0.84)
- PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement (NeurIPS 2025 · similarity 0.84)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.