Research map / Image generation / Image restoration
CAMSIC: Content-aware Masked Image Modeling Transformer for Stereo Image Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMSIC, a stereo image codec whose decoder-free Transformer entropy model uses content-aware masked image modeling (prior-derived tokens in place of [MASK]) to capture spatial and disparity dependencies.
Problem
Existing learned stereo image codecs use simple CNN entropy models adapted from single-image codecs, which fail to capture spatial-disparity dependencies well, especially for large disparities, yielding suboptimal rate-distortion.
Approach
Replace uninformative [MASK] tokens in masked image modeling with content-aware tokens generated from prior information (disparity prior and hyperprior), enabling bidirectional prior-token interaction inside a decoder-free, encoder-only Swin Transformer entropy model with iterative non-autoregressive decoding.
Key results
- On Cityscapes, CAMSIC achieves -41.961% BD-rate (PSNR) relative to BPG, versus -25.196% for ELIC and -36.665% for ECSIC.
- On InStereo2K, CAMSIC achieves -43.608% BD-rate (PSNR) vs -29.748% for ELIC and -43.551% for ECSIC.
- Ablation: replacing the proposed entropy model with the vanilla content-irrelevant MIM model raises BD-rate by 36.65% (Cityscapes) and 19.48% (InStereo2K) relative to the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CAMSIC entropy model with ECSIC entropy model (ablation) | CAMSIC · Cityscapes | BD-rate: 0% -> 48.89% | Hurt |
| replace CAMSIC entropy model with ECSIC entropy model (ablation) | CAMSIC · InStereo2K | BD-rate: 0% -> 22.38% | Hurt |
| replace CAMSIC entropy model with SASIC entropy model (ablation) | CAMSIC · Cityscapes | BD-rate: 0% -> 36.16% | Hurt |
| replace CAMSIC entropy model with SASIC entropy model (ablation) | CAMSIC · InStereo2K | BD-rate: 0% -> 19.44% | Hurt |
| replace CAMSIC entropy model with BCSIC entropy model (ablation) | CAMSIC · Cityscapes | BD-rate: 0% -> 5.36% | Hurt |
| replace CAMSIC entropy model with BCSIC entropy model (ablation) | CAMSIC · InStereo2K | BD-rate: 0% -> 5.76% | Hurt |
| replace content-aware MIM with vanilla content-irrelevant MIM entropy model (ablation) | CAMSIC · Cityscapes | BD-rate: 0% -> 36.65% | Hurt |
| replace content-aware MIM with vanilla content-irrelevant MIM entropy model (ablation) | CAMSIC · InStereo2K | BD-rate: 0% -> 19.48% | Hurt |
| add relative position embedding (ablation) | CAMSIC without RPE or ID · Cityscapes | BD-rate: 3.700 -> 2.739 | Helped |
| add relative position embedding (ablation) | CAMSIC without RPE or ID · InStereo2K | BD-rate: 1.316 -> 0.501 | Helped |
| add two identity embeddings (ablation) | CAMSIC without ID · Cityscapes | BD-rate: 2.739 -> 0% | Helped |
| add two identity embeddings (ablation) | CAMSIC without ID · InStereo2K | BD-rate: 0.501 -> 0% | Helped |
Surprising findings
- The vanilla MIM-based entropy model is inferior to the BCSIC bidirectional entropy model on the reported ablation, showing naive MIM transfer is insufficient.
Limitations
- Relies on the pretrained single-image codec ELIC for encoder/decoder to speed up experiments.
- Paper's ECSIC comparison requires training on large-width stereo images, and MV-HEVC only supports YUV420 which weakens that baseline.
Experimental setup
Datasets: Cityscapes InStereo2K
Benchmarks: Cityscapes InStereo2K
Metrics: BD-rate BD-PSNR BD-MS-SSIM bpp encoding/decoding time
Baselines: BPG ELIC HEVC VVC MV-HEVC LDMIC DispSIC SASIC BCSIC ECSIC
Builds on: ELIC MaskGIT Swin Transformer hyperprior ECSIC
Closest papers by meaning
- 3D-LMVIC: Learning-based Multi-View Image Compression with 3D Gaussian Geometric Priors (ICML 2025 · similarity 0.85)
- Low-Latency Neural Stereo Streaming (CVPR 2024 · similarity 0.83)
- Adaptive Multi-Modal Cross-Entropy Loss for Stereo Matching (CVPR 2024 · similarity 0.83)
- Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression (CVPR 2025 · similarity 0.83)
- Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec (NeurIPS 2024 · similarity 0.83)
- When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.