Research map / Image generation / Image restoration
Frequency-Aware Transformer for Learned Image Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FTIC, a learned image compression model with a frequency-aware transformer block (diverse-shaped window attention for low/high/vertical/horizontal frequencies plus a learnable Fourier filter FFN) and a transformer-based channel-wise autoregressive entropy model, reporting ~14.5% BD-rate savings over VTM-12.1 on Kodak.
Problem
Existing learned image compression models produce redundant latent representations because they fail to capture anisotropic (directional, multiscale) frequency components, and their entropy models either are costly or do not exploit channel dependencies well.
Approach
A frequency-aware transformer (FAT) block whose frequency-decomposition window attention uses four differently shaped windows (4s×4s, s×s, s×4s, 4s×s) to capture low, high, vertical and horizontal components in parallel, followed by a frequency-modulation FFN that applies learnable filters in the block-wise FFT domain; plus a T-CA entropy model that uses masked slice-wise channel attention.
Key results
- Outperforms VTM-12.1 by 14.5% (Kodak), 15.1% (Tecnick) and 13.0% (CLIC) in BD-rate.
- Full FAT block ablation: baseline -9.3% vs FAT block -14.5% BD-rate on Kodak (VTM-12.1 anchor).
- T-CA with 5 slices achieves -19.2% BD-rate (BPG anchor) at 30.4M params, versus CHARM (n_s=10) at -15.9% with 34.8M params.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard 4×4 window attention and conventional FFN with a larger 16×16 window (ablation) | FAT block · Kodak | BD-rate vs VTM-12.1: -9.3% -> -11.2% | Helped |
| Add FDWA instead of standard window attention (ablation) | FAT block · Kodak | BD-rate vs VTM-12.1: -9.3% -> -13.3% | Helped |
| Add FMFFN to the FDWA variant (ablation) | FAT block · Kodak | BD-rate vs VTM-12.1: -13.3% -> -14.5% | Helped |
| Replace CHARM with T-CA using 5 slices | mbt2018-mean transforms · Kodak | BD-rate vs BPG: -14.2% -> -19.2% | Helped |
| Increase T-CA transformer layers from 4 to 12 (ablation) | T-CA · Kodak | BD-rate vs BPG: -17.6% -> -19.2% | Helped |
| Increase T-CA transformer layers from 12 to 16 (ablation) | T-CA · Kodak | BD-rate vs BPG: -19.2% -> -19.0% | Hurt |
| Vary the number of T-CA slices from 5 to 4, 8, or 10 (ablation) | T-CA · Kodak | BD-rate vs BPG: -19.2%; -19.0%; -18.0%; -18.4% | Mixed |
Surprising findings
- Increasing the number of slices in T-CA does not consistently improve performance (n_s=4 and 5 are close; 8 and 10 are worse).
Limitations
- FDWA currently supports only two directions (horizontal and vertical), not richer multi-directional analysis.
- Focuses only on image compression; video extension left to future work.
Experimental setup
Datasets: Flickr2W ImageNet-1k Kodak Tecnick CLIC Professional Validation
Benchmarks: Kodak Tecnick CLIC Professional Validation
Metrics: BD-rate PSNR MS-SSIM bits per pixel GMACs parameters inference latency
Baselines: BPG VTM-12.1 CHARM STF mbt2018-mean
Builds on: Swin Transformer channel-wise autoregressive entropy model joint autoregression and hyperprior CHARM STF
Closest papers by meaning
- Learned Image Compression via Sparse Attention and Adaptive Frequency (CVPR 2026 · similarity 0.92)
- Learned Image Compression With Mixed Transformer-CNN Architectures (CVPR 2023 · similarity 0.88)
- Few-Shot Domain Adaptation for Learned Image Compression (AAAI 2025 · similarity 0.87)
- On Disentangled Training for Nonlinear Transform in Learned Image Compression (ICLR 2025 · similarity 0.87)
- Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning (ICLR 2025 · similarity 0.86)
- Frequency-Biased Synergistic Design for Image Compression and Compensation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.