Research map / Video / Video-language understanding

CoRDS: Coreset-Based Representative and Diverse Selection for Streaming Video Understanding

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoRDS, a training-free, query-agnostic KV-cache compressor for streaming video VLMs that selects a representative coreset via D²-style farthest-first selection in joint key-value space with an orthogonal anti-redundancy bonus.

Problem

Streaming video understanding with VLMs needs a bounded KV cache over unbounded visual history, but existing compressors use local token-level heuristics (recency, redundancy, norm saliency) that do not ensure the retained cache represents the full accumulated history.

Approach

Formulates streaming KV-cache compression as representative coreset selection in joint key-value space, using a bicriteria distance (weighted K and V squared distances) optimized by greedy D²-style farthest-first selection, plus a lightweight orthogonal residual bonus for diversity, with layer-selective compression and cross-layer index reuse.

Key results

• At 6K budget on Qwen2-VL-7B, CoRDS reaches 70.1 on MLVU vs 65.8 for Full KV and 65.9 for StreamMem (+4.2 over closest compressed competitor).
• On OVO-Bench at 6K budget with Qwen2-VL-7B, CoRDS overall 53.76 vs 53.34 for InfiniPot-V; StreamingBench average 73.43 vs 72.75.
• Efficiency: CoRDS matches InfiniPot-V peak VRAM (10.1-11.8 GB) across 300-1000 s videos where Full KV OOMs beyond 500 s, with higher prefill FPS (45.9 vs 44.4 at 1000 s).

Surprising findings

• Compressed caches sometimes outperform the uncompressed Full KV reference, e.g. CoRDS at 6K budget beats Full KV on MLVU for Qwen2-VL-7B.
• Layer selection matters: all-layer orthogonality fails to help, and compressing only the bottom 25% of layers gave the best accuracy in the ablation.

Limitations

• Gains on several benchmarks are small (e.g., +0.4 to +1.9 points on some EgoSchema and VideoMME rows), and hyperparameters were tuned on a held-out 200-sample MLVU PlotQA slice.
• Orthogonal term carries no approximation guarantee for the combined score; the theory covers only the orthogonal residual greedy rule.
• Query-agnostic design gives no mechanism for late-stage query conditioning, which the authors list as future work.

Experimental setup

Datasets: MLVU VideoMME EgoSchema OVO-Bench StreamingBench

Benchmarks: EgoSchema MLVU VideoMME OVO-Bench StreamingBench

Metrics: accuracy peak VRAM prefill FPS decode throughput coverage CDF distance

Baselines: InfiniPot-V StreamMem STC TTC SnapKV FastV Uniform InfiniPot

Builds on: D2 sampling InfiniPot-V StreamMem log-determinant subset selection

Closest papers by meaning

• What Should a Streaming Video Model Remember? (NeurIPS 2026 · similarity 0.91)
• Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.89)
• InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding (NeurIPS 2025 · similarity 0.89)
• BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.88)
• Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning (AAAI 2026 · similarity 0.88)