Research map / Video / Video segmentation and tracking
Towards Practical Real-Time Neural Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCVC-RT, a real-time neural video codec that cuts operational (non-compute) cost via implicit temporal modeling, single 1/8-resolution latents, model integerization, and a module-bank rate control, reaching 1080p 125/113 fps on A100 with 21% BD-rate savings over VTM.
Problem
Neural video codecs have strong rate-distortion performance but are too slow for real-time 1080p coding on consumer hardware, partly because computational cost alone is a poor predictor of speed and operational costs dominate; they also lack cross-device consistency and flexible rate control.
Approach
Reduce operational complexity (module count, latent size and memory I/O) rather than MACs: remove explicit motion estimation/compensation in favor of implicit temporal context, learn all latents at a single 1/8 resolution instead of progressive downsampling, and add integer-only inference plus a module bank of rate-adjustment vectors.
Key results
- DCVC-RT (fp16) achieves 125.2 fps encoding and 112.8 fps decoding for 1080p on an A100, with 21.0% average BD-rate savings over VTM-17.0 in YUV420.
- On an RTX 2080Ti, DCVC-RT reaches about 40 fps encoding and 34 fps decoding at 1080p.
- DCVC-RT matches DCVC-FM's compression (-21.0% vs -21.3% BD-rate) while being over 20x faster; int16 mode retains 18.3% savings over VTM but runs slower than fp16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use implicit temporal modelling instead of explicit motion modules (ablation) | DCVC-RT · MCL-JCV, large motion | BD-Rate: 0.0% -> 2.1% | Hurt |
| use implicit temporal modelling instead of explicit motion modules (ablation) | DCVC-RT · MCL-JCV, small motion | BD-Rate: 0.0% -> –0.4% | Helped |
| use implicit temporal modelling instead of explicit motion modules (ablation) | DCVC-RT · MCL-JCV, scene change | BD-Rate: 0.0% -> –4.7% | Helped |
| use implicit temporal modelling instead of explicit motion modules (ablation) | DCVC-RT · MCL-JCV | encoding time: 27.2 ms -> 8.0 ms | Helped |
| learn latents at a single 1/8 resolution instead of progressive downsampling (ablation) | DCVC-RT · video compression | BD-Rate: 0.3% | No clear effect |
| use a rate-adjustment module bank instead of a single factorized prior for z (ablation) | DCVC-RT · UVG | bits: 3.4% bits saving | Helped |
| integerize the model to int16 instead of using fp16 (ablation) | DCVC-RT · average benchmark | BD-Rate (%): −21.0% -> –18.3% | Hurt |
| integerize the model to int16 instead of using fp16 (ablation) | DCVC-RT · 1080p, NVIDIA A100 | coding speed: 125.2 / 112.8 -> 28.3 / 20.9 | Hurt |
| use DCVC-RT (fp16) instead of VTM-17.0 | 1080p video compression · Average | BD-Rate (%): 0.0 -> −21.0 | Helped |
| use DCVC-RT (fp16) instead of DCVC-FM | 1080p video compression · Average | BD-Rate (%): −22.1 -> −21.0 | No clear effect |
Surprising findings
- Reducing channel count yields roughly linear rather than quadratic speedup, indicating operational rather than computational cost is the main bottleneck.
- int16 integerized inference is significantly slower than fp16 on modern GPUs due to lack of int16 optimization (e.g., over 4x slower on A100).
Limitations
- Performance drops in the high-quality range (above about 40 dB) due to the lightweight model design.
- int16 integerized mode is slower than fp16 because modern GPUs lack int16 optimization.
Experimental setup
Datasets: Vimeo-90k Vimeo original videos
Benchmarks: HEVC Class B-E UVG MCL-JCV
Metrics: BD-Rate encoding fps decoding fps MACs params
Baselines: VTM-17.0 HM-16.25 ECM-11.0 DCVC-DC DCVC-FM x265 DHVC-2.0 C3 MobileNVC ELF-VC
Builds on: DCVC-FM DCVC-DC conditional coding compressive autoencoder
Closest papers by meaning
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.91)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.91)
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.90)
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.90)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.88)
- C3: High-Performance and Low-Complexity Neural Compression from a Single Image or Video (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.