Research map / Vision-language and multimodal / Cross-modal learning
End-to-End RGB-D Image Compression via Exploiting Channel-Modality Redundancy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a learning-based dual-branch RGB-D image compression network that codes in YUV420 with intra- and cross-modality attention and a conditional context entropy model, outperforming BPG and learned baselines on SUN-RGBD and NYU-Depth V2.
Problem
Existing learned image compression methods code RGB and depth images separately, failing to exploit redundancy between modalities and channels in RGB-D images, which limits rate-distortion performance.
Approach
A dual-branch transformer codec operating in YUV domain, with Intra-Modality Attention (self-attention across channels within a modality), Cross-Modality Attention (cross-attention between RGB and depth latents), and a conditional context entropy model with Context Prediction and Context Fusion modules that use cross-modal priors.
Key results
- Proposed method achieves -15.717% BD-Rate vs BPG on RGB images of SUN-RGBD and -12.376% on NYU-Depth V2 RGB.
- Ablation: conditional context entropy model gives -7.56% BD-Rate; IMA+CMA together give -6.71% over baseline.
- Model has 69.03M parameters; encoding 11.696 s and decoding 8.582 s per image on Tesla V100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed RGB-D compression framework instead of SASIC | RGB-D image compression · SUN-RGBD RGB | BD-Rate (%): -9.310 -> -15.717 | Helped |
| Use the proposed RGB-D compression framework instead of SASIC | RGB-D image compression · SUN-RGBD depth | BD-Rate (%): -25.918 -> -36.244 | Helped |
| Use the proposed RGB-D compression framework instead of SASIC | RGB-D image compression · NYU-Depth V2 RGB | BD-Rate (%): -8.118 -> -12.376 | Helped |
| Use the proposed RGB-D compression framework instead of SASIC | RGB-D image compression · NYU-Depth V2 depth | BD-Rate (%): -30.462 -> -38.971 | Helped |
| Add Conditional Context-based Entropy Model (ablation) | Mbt2018 baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Add Context Prediction Module (CPM) (ablation) | Mbt2018 baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Add Context Fusion Module (CFM) (ablation) | Mbt2018 baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Add Intra-Modality Attention (IMA) (ablation) | RGB-D compression baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Add Cross-Modality Attention (CMA) (ablation) | RGB-D compression baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Add both IMA and CMA (ablation) | RGB-D compression baseline · SUN-RGBD | BD-Rate (%): - | Helped |
| Compress in the YUV domain instead of the RGB domain (ablation) | RGB-D compression framework with IMA and CMA · RGB-D images | BD-Rate (%) | Helped |
Surprising findings
- Depth images show larger gains over BPG (-36.2% on SUN-RGBD) than RGB images, while many single-image learned baselines are worse than BPG on depth.
Limitations
- High computational cost (69M parameters, ~12 s encoding) compared with other learned codecs.
- Relies on fine-tuning for NYU-Depth V2 and training mainly on SUN-RGBD.
- Evaluated only on two RGB-D datasets with PSNR metric.
Experimental setup
Datasets: SUN-RGBD NYU-Depth V2
Benchmarks: SUN-RGBD NYU-Depth V2
Metrics: PSNR BD-Rate bits per pixel
Baselines: Factorized Hyperprior Mbt2018 Cheng2020 WBA TIC Coarse2fine SASIC BPG
Builds on: Mbt2018 Swin Transformer CompressAI Cheng2020 attention
Closest papers by meaning
- Learning Based Multi-Modality Image and Video Compression (CVPR 2022 · similarity 0.87)
- Deep RGB-D Saliency Detection With Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR 2021 · similarity 0.84)
- Multi-Modal Contrastive Masked Autoencoders: A Two-Stage Progressive Pre-training Approach for RGBD Datasets (CVPR 2025 · similarity 0.82)
- Linear Attention Modeling for Learned Image Compression (CVPR 2025 · similarity 0.82)
- Visual Redundancy Removal for Composite Images: A Benchmark Dataset and a Multi-Visual-Effects Driven Incremental Method (AAAI 2024 · similarity 0.82)
- GLIC: General Format Learned Image Compression (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.