Research map / Video / Video segmentation and tracking
Combining Frame and GOP Embeddings for Neural Video Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-NeRV, a hybrid video implicit neural representation combining per-frame ConvNeXt embeddings with a multi-resolution GOP feature grid and a flow-based decoder, trained with entropy-constrained compression to outperform prior video INRs and HEVC on UVG.
Problem
Existing implicit neural representation (INR) video methods perform well only in limited settings (specific model sizes, fixed aspect ratios, low-motion or high-motion content) and lag behind traditional codecs such as HEVC in compression.
Approach
A hybrid frame-wise INR that fuses large frame-specific embeddings (from a big ConvNeXt-based encoder, temporally modulated) with GOP-specific features from a multi-resolution feature grid, decoded by an optical-flow-based decoder; the ratio between the two acts as a tunable lever, and end-to-end entropy-constrained training lets the model allocate bits to content-specific components.
Key results
- On UVG at 720p with 12.4M params, T-NeRV averages 38.12 dB PSNR versus 37.10 for FFNeRV, the second-best method.
- On the 2:1 setting (1920x960, 3.0M params), T-NeRV averages 34.00 dB versus 32.66 dB for HNeRV.
- In compression, T-NeRV is reported as the first video INR to outperform HEVC (x265 veryslow, no b-frames) on all UVG sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine frame-specific embeddings with GOP-specific features | T-NeRV · UVG, 720p, 12.4M parameters (Avg.) | PSNR: 37.10 -> 38.12 | Helped |
| combine frame-specific embeddings with GOP-specific features | T-NeRV · UVG, 2:1 crop, 3.0M parameters (Avg.) | PSNR: 32.67 -> 34.00 | Helped |
| use frame-specific embeddings only instead of combining embeddings and GOP-specific feature grids (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 32.23 -> 32.45 | Helped |
| use GOP-specific feature grids only instead of combining embeddings and feature grids (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.21 -> 32.45 | Helped |
| add temporal augmentation to frame-specific embeddings (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.52 -> 32.45 | Helped |
| use the larger ConvNeXt-based encoder instead of HNeRV's tiny encoder (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.52 -> 32.45 | Helped |
| use the larger ConvNeXt-based encoder and retain temporal injection instead of the tiny encoder without temporal injection (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.09 -> 32.45 | Helped |
| add flow-guided frame aggregation (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.45 -> 32.45 | Helped |
| add temporal information injection via AdaIN (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.30 -> 32.45 | Helped |
| replace ordinary 5×5 convolutions with modified T-NeRV blocks (ablation) | T-NeRV · UVG, 720p, first 300 frames, 3M parameters (Ready.) | PSNR: 31.48 -> 32.45 | Helped |
| entropy-constrained training allocates more bits to GOP-specific features for high-motion content | T-NeRV · UVG, 600-frame sequences | compressed video size | Mixed |
Surprising findings
- Entropy-constrained training automatically allocates more bits to feature grids for high-motion videos and to embeddings for static ones, without manual tuning.
Limitations
- Encoding is slow, requiring many hours of training per sequence.
- Evaluated only on the UVG dataset.
- Entropy estimation cost limits speed of the compression pipeline.
Experimental setup
Datasets: UVG
Benchmarks: UVG
Metrics: PSNR MS-SSIM BPP BD-rate
Baselines: NeRV E-NeRV C-NeRV FFNeRV HNeRV DNeRV HEVC (x265 veryslow)
Builds on: NeRV HNeRV FFNeRV C-NeRV (Gomes et al.) ConvNeXt
Closest papers by meaning
- Boosting Neural Representations for Videos with a Conditional Decoder (CVPR 2024 · similarity 0.91)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.90)
- DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes (CVPR 2024 · similarity 0.89)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.88)
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.87)
- MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.