Research map / Architectures and efficiency / Attention and transformers
Linear Attention Modeling for Learned Image Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LALIC, a learned image compression codec built from linear-attention RWKV blocks (Bi-RWKV) and an RWKV-based spatial-channel context entropy model, reporting about 15-18% BD-rate savings over VTM-9.1.
Problem
Learned image compression gains have come with growing computational cost, and transformer backbones are quadratic in sequence length, so a low-complexity transform and entropy model is needed for high-resolution images.
Approach
Replace transformer and CNN transform blocks with Bi-RWKV blocks (spatial-mix using bidirectional WKV linear attention plus channel-mix, with Omni-Shift for 2D local context), and use an RWKV-based spatial-channel context model for entropy coding.
Key results
- BD-rate over VTM-9.1 of -15.26% on Kodak, -15.41% on CLIC and -17.63% on Tecnick (PSNR).
- Ablation: RWKV-SCCTX entropy model reaches -3.50% BD-rate vs. -2.74% for Conv Plus SCCTX at similar parameter count (63.24M vs 62.08M) and fewer FLOPs.
- Decoding time 150 ms and 286.16 GFLOPs on Kodak, with 63.24M parameters, on an RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LALIC instead of VTM-9.1 | Image compression · Kodak | BD-rate (PSNR): 0.00% -> -15.26% | Helped |
| Use LALIC instead of VTM-9.1 | Image compression · CLIC | BD-rate (PSNR): 0.00% -> -15.41% | Helped |
| Use LALIC instead of VTM-9.1 | Image compression · Tecnick | BD-rate (PSNR): 0.00% -> -17.63% | Helped |
| Use 2,4,6,2 Bi-RWKV layers instead of 2,2,2,2 (ablation) | LALIC with Conv SCCTX · Kodak | BD-rate: 0.00% -> -2.50% | Helped |
| Increase Bi-RWKV layers from 2,4,6,2 to 2,4,6,6 (ablation) | LALIC with Conv SCCTX · Kodak | BD-rate: -2.50% -> -1.68% | Hurt |
| Replace Conv SCCTX with Conv Plus SCCTX (ablation) | LALIC · Kodak | BD-rate: -1.68% -> -2.74% | Helped |
| Replace Conv Plus SCCTX with RWKV-SCCTX (ablation) | LALIC · Kodak | BD-rate: -2.74% -> -3.50% | Helped |
| Add Shift to AFT attention (ablation) | LALIC Bi-RWKV block · Kodak | test R-D loss: 0.5657 -> 0.5604 | Helped |
| Replace AFT with BiWKV attention (ablation) | LALIC Bi-RWKV block with Shift · Kodak | test R-D loss: 0.5604 -> 0.5551 | Helped |
Surprising findings
- Adding more Bi-RWKV blocks in the transform (2,4,6,2 vs 2,2,2,2) reduced BD-rate gain only modestly in a stacked comparison, and the 2,4,6,6 Conv variant was slightly worse than 2,4,6,2 in Table 2.
Limitations
- Encoding time (274 ms) is higher than several baselines and the paper reports decoding and encoding only for one GPU.
- FAT decoding was re-run and reported as over 10 s in the table, so comparisons depend on re-run conditions.
- Gains are reported only for PSNR-optimized BD-rate in the main table; MS-SSIM results are shown only as curves.
Experimental setup
Datasets: OpenImages (first 400k images)
Benchmarks: Kodak CLIC Professional Validation Tecnick
Metrics: BD-rate (PSNR) BD-rate (MS-SSIM) PSNR MS-SSIM BPP encoding time decoding time GPU memory FLOPs parameters
Baselines: VTM-9.1 BPG ELIC MambaVC TCM-large FAT MLIC++ Minnen20 Cheng20-Parallel
Builds on: RWKV Vision-RWKV Omni-Shift Attention-Free Transformer Cheng2020 checkerboard/channel context model
Closest papers by meaning
- GLIC: General Format Learned Image Compression (AAAI 2025 · similarity 0.88)
- Another Way to the Top: Exploit Contextual Clustering in Learned Image Coding (AAAI 2024 · similarity 0.87)
- FLAVC: Learned Video Compression with Feature Level Attention (CVPR 2025 · similarity 0.87)
- MambaIC: State Space Models for High-Performance Learned Image Compression (CVPR 2025 · similarity 0.86)
- Learned Image Compression with Dictionary-based Entropy Model (CVPR 2025 · similarity 0.86)
- Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.