Research map / Video / Video-language understanding
ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streams
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReSpec, an online filtering framework for video-text streams that combines CLIP-style alignment thresholding, vMF kernel-density relevance testing against downstream tasks, and root-embedding specificity, training BT-Adapter with ~27% (WebVid2M) or ~5% (VideoCC3M) of data.
Problem
Video-text data streams are too large to store and train on fully; online training needs to select data that is clean, relevant to downstream tasks, and informative, at low compute cost.
Approach
Filters each incoming video-text pair in real time using three cheap checks on pretrained LanguageBind embeddings: cross-modal similarity threshold, a statistical relevance test via von Mises-Fisher kernel density against downstream task embeddings, and a specificity test using distance from the empty-text root embedding.
Key results
- ReSpec reaches average recall 49.11 on WebVid2M using 27.5% of data, versus 47.23 for full-data training and 48.17 for alignment-only filtering.
- On VideoCC3M, ReSpec uses about 5% of the data and achieves the best average performance among filtering baselines (CiT, CoLoR-Filter, LB/CLIP thresholds).
- Text-modality relevance filtering outperforms video-only relevance (49.11 vs 47.99 average recall on WebVid2M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multimodal alignment filtering (ablation) | ReSpec online filtering on WebVid2M · WebVid2M | Avg. perf.: 47.23 -> 48.17 | Helped |
| Add relevance filtering after alignment filtering (ablation) | ReSpec online filtering on WebVid2M · WebVid2M | Avg. perf.: 48.17 -> 48.79 | Helped |
| Add specificity filtering after alignment and relevance filtering (ablation) | ReSpec online filtering on WebVid2M · WebVid2M | Avg. perf.: 48.79 -> 49.11 | Helped |
| Use text rather than video for relevance filtering (ablation) | ReSpec online filtering · WebVid2M | Avg. perf.: 47.99 -> 49.11 | Helped |
| Require either text or video to be relevant instead of using text relevance (ablation) | ReSpec online filtering · WebVid2M | Avg. perf.: 49.11 -> 48.10 | Hurt |
| Require both text and video to be relevant instead of using text relevance (ablation) | ReSpec online filtering · WebVid2M | Avg. perf.: 49.11 -> 48.94 | Hurt |
| Use kernel density estimation rather than a single von Mises-Fisher distribution (ablation) | ReSpec relevance filtering · WebVid2M | Avg. perf.: 48.59 -> 49.11 | Helped |
| Set the specificity quantile to q = 0.1 rather than q = 0.05 (ablation) | ReSpec specificity filtering · WebVid2M | Avg. perf.: 48.55 -> 49.11 | Helped |
| Set the specificity quantile to q = 0.15 rather than q = 0.1 (ablation) | ReSpec specificity filtering · WebVid2M | Avg. perf.: 49.11 -> 48.94 | Hurt |
| Set the specificity quantile to q = 0.25 rather than q = 0.1 (ablation) | ReSpec specificity filtering · WebVid2M | Avg. perf.: 49.11 -> 48.88 | Hurt |
| Set the specificity quantile to q = 0.50 rather than q = 0.1 (ablation) | ReSpec specificity filtering · WebVid2M | Avg. perf.: 49.11 -> 48.83 | Hurt |
Surprising findings
- Filtering with a single vMF distribution instead of kernel density estimation oversamples data and lowers performance (48.59 vs 49.11).
- Video-only relevance filtering oversamples (63.7% of data kept) and performs worse than text-based relevance.
Limitations
- Relies on a pretrained feature extractor (LanguageBind) and a fixed alignment threshold tau, with results reported as best over tuned thresholds.
- Evaluated only on two web-scale noisy video-text sources and five zero-shot retrieval tasks with a single online model (BT-Adapter).
Experimental setup
Datasets: WebVid2M VideoCC3M MSR-VTT DiDeMo LSMDC ActivityNet YouCook2
Benchmarks: MSR-VTT DiDeMo LSMDC ActivityNet YouCook2
Metrics: Recall@1 Recall@5 Recall@10 average recall clip ratio Frechet distance KL divergence
Baselines: CLIP Threshold LB Threshold CiT CoLoR-Filter full data training
Builds on: CiT CoLoR-Filter HYPE BT-Adapter DSIR
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.84)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.84)
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval (NeurIPS 2025 · similarity 0.84)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.84)
- DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval (AAAI 2024 · similarity 0.83)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.