Research map / Image generation / Image restoration
CosAE: Learnable Fourier Series for Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CosAE, an autoencoder whose extremely narrow bottleneck stores learnable Fourier amplitudes/phases that are expanded into 2D cosine basis functions, and applies it to flexible-resolution super-resolution and blind face/natural image restoration.
Problem
Conventional autoencoders with spatially downsampled bottlenecks lose high-frequency detail, so they are rarely used directly for pixel-level image restoration; the paper seeks a generic autoencoder that keeps detail under extreme compression and generalizes across unknown degradations and upscaling
Approach
The bottleneck encodes learnable amplitude and phase coefficients rather than spatial features; a harmonic construction module expands them with learnable, globally shared cosine frequencies into T×T harmonic maps that a decoder sums to reconstruct the image, enabling 32-64x downsampling without detail loss and resolution-agnostic inference by varying T.
Key results
- On FFHQ+CelebA FR-SR at 8x (LR 32), CosAE (G) reaches FID 12.81 and PSNR 23.65 versus LIIF-64x (G) at 18.91 FID and 22.59 PSNR.
- On CelebA-Test blind face restoration, CosAE achieves PSNR 25.65, SSIM 0.70 and LPIPS 0.27, the best among compared methods, though its FID (52.03) is worse than GFP-GAN and RestoreFormer.
- On ImageNet 4x super-resolution, CosAE reports FID-50k 2.1 with 67M parameters versus LDM-4 at 2.8 FID and 169M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CosAE rather than LIIF-64x (G) | FR-SR face models · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.30 -> 0.24 | Helped |
| Use CosAE rather than LIIF-4x (G) | FR-SR natural-image models · DIV2K val, up-ratio ×8 | LPIPS: 0.41 -> 0.33 | Helped |
| Use CosAE rather than RestoreFormer | Blind face restoration · CelebA-Test | PSNR: 24.42 -> 25.65 | Helped |
| Use CosAE rather than ResShift | ×4 super-resolution models · ImageNet-val | LPIPS: 0.126 -> 0.167 | Hurt |
| Use CosAE rather than SCUNet | Natural-image restoration · COCO-Test synthetic corruption | FID-5k: 45.47 -> 22.34 | Helped |
| Replace Cosine basis functions with plain upsampling layers (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.24 -> 0.31 | Hurt |
| Encode the input with Cosine basis functions and use the nocos bottleneck (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.24 -> 0.31 | Hurt |
| Sum frequencies in the bottleneck before the decoder (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.24 -> 0.26 | Hurt |
| Fix frequencies to uniform samples rather than learning them (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.24 -> 0.28 | Hurt |
| Initialize frequencies with low frequencies only (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | PSNR (db): 24.34 -> 23.99 | Hurt |
| Mask aliased basis functions during training (ablation) | CosAE · FFHQ+CelebA val, LR-res 32 | LPIPS: 0.24 -> 0.25 | Hurt |
Surprising findings
- A 64x downsampled bottleneck can outperform wider-bottleneck LIIF variants on flexible-resolution super-resolution, which is counter to the common belief that no bottleneck is best for restoration.
- Strictly constraining learned frequencies to avoid aliasing is not essential; the non-aliased variant (-NoAls) matches or improves FID.
Limitations
- Out-of-distribution upsampling by increasing T at inference is unreliable because high frequencies above Tmax/2 were never learned.
- Face model generalizes poorly to high-resolution faces not seen in training; artifacts appear as sinusoidal-like waves on extremely low-resolution or degraded inputs.
- Face FID is worse than several baselines (CodeFormer, GFP-GAN, RestoreFormer), so realism is not uniformly best.
Experimental setup
Datasets: FFHQ CelebA-HQ DIV2K Flickr2K ImageNet COCO
Benchmarks: FFHQ+CelebA val (FR-SR) DIV2K val CelebA-Test LFW-Test WebPhoto-Test WIDER-Test ImageNet-val COCO-Test RNI15
Metrics: FID LPIPS PSNR SSIM IDD MUSIQ
Baselines: LIIF LTE ITNSR DFDNet PULSE GFP-GAN RestoreFormer CodeFormer LDM-4 SCUNet
Builds on: VQVAE LIIF LTE CodeFormer
Closest papers by meaning
- DreamUHD: Frequency Enhanced Variational Autoencoder for Ultra-High-Definition Image Restoration (AAAI 2025 · similarity 0.85)
- Improving the Learning Capability of Small-size Image Restoration Network by Deep Fourier Shifting (NeurIPS 2024 · similarity 0.85)
- UniDemoiré: Towards Universal Image Demoiréing with Data Generation and Synthesis (AAAI 2025 · similarity 0.83)
- Omni-Kernel Network for Image Restoration (AAAI 2024 · similarity 0.83)
- AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation (ICLR 2025 · similarity 0.83)
- Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.