Research map / Video / Video-language understanding
RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RELOCATE, a training-free region-based pipeline (SAM segmentation, DINO features, cosine search, refinement, SAM 2 tracking, query expansion) for visual query localization in long egocentric videos, outperforming trained methods on Ego4D VQ2D.
Problem
Visual query localization: localizing the last appearance of a query object (given as a reference image) in long untrimmed videos, where prior methods require task-specific training and must re-encode the video per query.
Approach
Uses a classic stage-wise object localization pipeline built on region-based object tokens (SAM masks pooled over DINO features) that are query-independent and reusable, adds object-centric crop refinement and multi-query expansion with reiteration, and tracks with SAM 2, all without training.
Key results
- On the VQ2D test set, RELOCATE reaches 0.35 stAP25 and 0.43 tAP25, a 49% and 33% relative improvement over the next-best baseline (VQLoC at 0.24 and 0.32).
- On the validation set, RELOCATE achieves 0.33 stAP25 versus 0.22 for VQLoC without any task-specific training.
- Ablations show removing refinement drops stAP25 from 0.333 to 0.246 and removing reiteration drops it to 0.247.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add search refinement using object-centric crops and rescoring (ablation) | RELOCATE · Ego4D VQ2D validation | stAP25: 0.246 -> 0.333 | Helped |
| Add query expansion and reiteration (ablation) | RELOCATE · Ego4D VQ2D validation | stAP25: 0.247 -> 0.333 | Helped |
| Use DINO ViT-B/8 instead of DINO ViT-B/16 (ablation) | RELOCATE · Ego4D VQ2D validation | stAP25: 0.254 -> 0.333 | Helped |
| Use DINO ViT-B/8 instead of DINOv2 ViT-L/14 (ablation) | RELOCATE · Ego4D VQ2D validation | tAP25: 0.452 -> 0.409 | Hurt |
| Use region-based rather than patch-based representations (ablation) | RELOCATE · VQ2D comparative analysis | spatial localization: 42% vs. 58% | Helped |
| Use region-based rather than patch-based representations (ablation) | RELOCATE · VQ2D comparative analysis | frame-level retrieval: 65% vs. 60% | Hurt |
| Use test-time training instead of cosine similarity search (ablation) | RELOCATE · VQ2D | retrieval performance | Hurt |
| Use DINOv2 ViT-L/14 instead of DINO ViT-B/8 (ablation) | RELOCATE · Ego4D VQ2D validation | tAP25: 0.409 -> 0.452 | Helped |
Surprising findings
- Region-based representations gave better spatial localization (58% vs 42%) than patch-based ones, though patch encodings gave better frame-level retrieval (65% vs 60%).
- Test-time training of the query matcher performed worse than simple cosine similarity search.
Limitations
- Video preparation is slow (about 1422 s per 1000 frames) and the implementation is not optimized for speed.
- Failures occur when the visual query is ambiguous or another object closely resembles the query.
Experimental setup
Datasets: Ego4D VQ2D
Benchmarks: Ego4D VQ2D
Metrics: stAP25 tAP25 success recovery
Baselines: SiamRCNN NFM CocoFormer VQLoC
Builds on: SAM DINO SAM 2 VQLoC region-based retrieval (SAM+DINOv2 pooling)
Closest papers by meaning
- Enhancing Visual Localization with Cross-Domain Image Generation (ICML 2025 · similarity 0.88)
- Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.88)
- QDETRv: Query-Guided DETR for One-Shot Object Localization in Videos (AAAI 2024 · similarity 0.87)
- GPVK-VL: Geometry-Preserving Virtual Keyframes for Visual Localization under Large Viewpoint Changes (CVPR 2025 · similarity 0.87)
- From Coarse to Fine: A Matching and Alignment Framework for Unsupervised Cross-View Geo-Localization (AAAI 2025 · similarity 0.87)
- Localizing Events in Videos with Multimodal Queries (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.