Research map / Video / Video-language understanding

OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OmniMem, an explicit full-range sparse KV-block retrieval framework for chunk-based autoregressive DiT video generation that fixes local bias and "Union Explosion" via adaptive window exclusion, query-shared selection, and per-head scattered KV access.

Problem

Long chunk-based autoregressive video generation requires access to a growing historical KV cache; truncation or compression loses query-relevant history, and naive sparse retrieval suffers from local bias and a large union of selected blocks across queries and heads.

Approach

Adapt NSA-style compression/selection/sliding-window attention to chunk-based video DiTs by excluding SWA-covered blocks from top-K candidates once enough history exists, sharing top-K selections across groups of adjacent queries, and letting each head gather its own non-contiguous KV blocks via a pointer table instead of a merged buffer.

Key results

• On 60-second VBench-Long generation, OmniMem reaches Dynamic Degree 82.29 vs. 53.47 for LongLive, a 52.3% improvement, with total score 83.08 (best) at 23.3 GB VRAM, 1.7% above LongLive.
• At 717 frames, OmniMem uses 3.3x less memory and runs 2.7x faster than full-history attention, matching Naive Sparse latency.
• On short 5-second VBench, OmniMem scores 84.29 total, nearly matching Self-Forcing (84.31) and achieving the highest Quality score (86.28).

Surprising findings

• Adding fixed sink tokens to OmniMem gives no additional gain, suggesting dynamic retrieval already captures the useful global context.
• Sharing head-level selections across heads (G_h=3 or 12) degrades all metrics, indicating heads attend to genuinely different historical regions.

Limitations

• Evaluated only on a single 1.3B backbone (Wan2.1-T2V-1.3B), so generalization to larger models is untested.
• Triton kernel tuned for current GPUs; latency and memory gains may vary across hardware.
• Long-video results rely on VBench-Long metrics with single-prompt setup; no human evaluation reported.

Experimental setup

Datasets: VidProM VBench VBench-Long

Benchmarks: VBench-Long VBench

Metrics: VBench total score quality score semantic score subject consistency imaging quality dynamic degree background consistency CLIP score VRAM

Baselines: LongLive MemFlow RollingForcing SkyReels-V2 Self-Forcing CausVid Pyramid Flow MAGI-1 NOVA Naive Sparse

Builds on: Native Sparse Attention Self-Forcing LongLive Wan2.1

Closest papers by meaning

• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.87)
• CoRDS: Coreset-Based Representative and Diverse Selection for Streaming Video Understanding (NeurIPS 2026 · similarity 0.86)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.86)
• Addressable Memory for Video World Models (NeurIPS 2026 · similarity 0.86)
• AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.86)
• StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.86)