Research map / Video / Video segmentation and tracking
RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a PPO-based RL agent that outputs per-macroblock QP deltas for the x264 encoder, optimizing downstream task performance (car detection, ROI encoding) under bit-rate constraints without ground-truth labels at inference.
Problem
Standard video encoders optimize for human perception, while many videos are consumed by AI systems. Task-aware compression typically needs downstream task information or heavy computation at encoding time, and is not differentiable when built on standard codecs.
Approach
Control macro-block QP deltas of a standard x264 encoder with an RL policy trained with a self-supervised task reward (task output on raw frame as pseudo-ground-truth), a bit-rate reward, and an auxiliary block-level reward prediction loss; the policy needs no task input at inference.
Key results
- Car detection BD-rate reduction of 24.7% (±1.38%) versus x264 with PSNR BD-rate cost of only 1.19%.
- ROI encoding saliency-weighted PSNR BD-rate of -25.64% (±0.99%) versus x264.
- Policy trained on YOLOv5-nano transfers to SSD precision (BD-rate -26.2%) and car segmentation IOU (-14.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RL-RC-DoT instead of x264 | Video compression for car detection · BDD100K test set, low-rate | precision: .22 ± .0013 -> .36± .0015 | Helped |
| Use RL-RC-DoT instead of x264 | Video compression for car detection · BDD100K test set, high-rate | recall: .81 ± .001 -> .83± .001 | Helped |
| Use RL-RC-DoT instead of x264 | Video compression for car detection · BDD100K test set, high-rate | PSNR: 34.55 ± .03 -> 34.55 ± .03 | No clear effect |
| Use RL-RC-DoT instead of x264 | Video compression for car detection · Car detection, YOLOv5-nano | precision BD-rate: −24.7 ± 1.57 | Helped |
| Use RL-RC-DoT instead of x264 | Video compression for car detection · Car detection, YOLOv5-nano | PSNR BD-rate: 1.19 ± 0.46 | Hurt |
| Use RL-RC-DoT instead of x264 | Video compression for ROI encoding · ROI encoding | saliency-weighted PSNR BD-rate: −25.64 ± 0.99 | Helped |
| Add macro-block reward information to training (ablation) | RL-RC-DoT · Car detection | precision BD-rate: −19.4 ± 1.38 -> −24.7 ± 1.57 | Helped |
| Add macro-block reward information to training (ablation) | RL-RC-DoT · ROI encoding | saliency-weighted PSNR BD-rate: −23.46 ± 0.97 -> −25.64 ± 0.99 | Helped |
| Use a long-term policy instead of setting γ = 0 (ablation) | RL-RC-DoT · Car detection | precision BD-rate: −9.78 ± 1.29 -> −24.7 ± 1.57 | Helped |
| Use a long-term policy instead of setting γ = 0 (ablation) | RL-RC-DoT · ROI encoding | saliency-weighted PSNR BD-rate: −16.01 ± 0.77 -> −25.64 ± 0.99 | Helped |
Surprising findings
- Removing the macro-block reward information only modestly degrades results, while the myopic gamma=0 variant loses most of the gain.
- ROI encoding yields a PSNR BD-rate improvement (-5.26%) even though the policy optimizes saliency-weighted PSNR, not plain PSNR.
Limitations
- Training slows down with downstream task complexity because encoding and task inference run per frame.
- Generalizing across video resolutions is hard because it changes the action space size.
- Only evaluated with H.264 (x264); extension to H.265/H.266 requires adaptation.
Experimental setup
Datasets: BDD100K
Benchmarks: BDD100K
Metrics: precision recall PSNR BD-rate saliency-weighted PSNR IOU
Baselines: x264 default rate control RL-RC-DoT w/o reward information RL-RC-DoT gamma=0
Builds on: PPO x264 Eigen-CAM
Closest papers by meaning
- RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation (NeurIPS 2025 · similarity 0.86)
- ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis (ICLR 2026 · similarity 0.85)
- Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder (AAAI 2025 · similarity 0.85)
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.84)
- ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding (NeurIPS 2025 · similarity 0.84)
- Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMs (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.