Research map / Image generation / GANs
E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E$^2$GAN, which makes distilling diffusion-model image editing into GANs efficient by pretraining a base GAN on many concepts, fine-tuning only LoRA on crucial layers with a rank search, and clustering-based data reduction.
Problem
Distilling large text-to-image diffusion models into lightweight GANs for on-device image-to-image editing requires costly training from scratch for every new concept.
Approach
Build a pretrained base GAN (3 ResNet blocks plus one transformer block) over many concepts, then adapt to new concepts by training only LoRA modules on crucial layers, with LoRA ranks chosen by a doubling search and training data reduced by K-means similarity clustering.
Key results
- E$^2$GAN achieves average FID 80.28 on FFHQ vs 86.03 for pix2pix and 109.37 on Flickr-Scenery vs 114.2 for pix2pix.
- E$^2$GAN updates only 0.092M parameters per new concept, 869x fewer than pix2pix-zero-distilled, with 14x/25x training FLOPs savings vs pix2pix/pix2pix-zero-distilled.
- The 3RB+1TB generator has 7.1M parameters and 15.5 ms latency on iPhone 14 Pro GPU, under the 33.3 ms real-time target.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use E²GAN instead of Pix2pix | GAN image-to-image translation · FFHQ | FID: 86.03 -> 80.28 | Helped |
| use E²GAN instead of Pix2pix-zero-distilled | GAN image-to-image translation · Landscape | FID: 132.6 -> 109.37 | Helped |
| use the 3RB+1TB architecture instead of Pix2pix's 9RB architecture (ablation) | image-to-image GAN trained from scratch · FFHQ concepts | FID: 49.14 -> 42.65 | Helped |
| use one transformer block instead of two (ablation) | E²GAN generator · Image-to-image generation | latency: 21.0 ms -> 15.5 ms | Helped |
| train the base model on 20 selected representative concepts rather than 20 random concepts (ablation) | E²GAN base model · FFHQ: White walker | FID: 53.92 -> 40.18 | Helped |
| train the base model on multiple representative concepts rather than a single concept (ablation) | E²GAN base model · FFHQ: White walker | FID: 51.99 -> 40.18 | Helped |
| use the searched LoRA ranks rather than upscaling each rank 1× (ablation) | E²GAN LoRA fine-tuning · Flickr Scenery | FID: 130.98 -> 109.37 | Helped |
| use the searched LoRA ranks rather than randomly assigning ranks (ablation) | E²GAN LoRA fine-tuning · Flickr Scenery | FID: 129.87 -> 109.37 | Helped |
| select training data with Similarity Clustering rather than random sampling (ablation) | E²GAN fine-tuning · FFHQ and Flickr Scenery concepts | FID | Helped |
| replace CONV and transpose CONV around the transformer block with Max Pool and Max Unpool (ablation) | E²GAN generator · Flickr Scenery: Forest in the dark | FID: 121.60 -> 190.05 | Hurt |
| remove cross-attention during fine-tuning (ablation) | E²GAN · FFHQ: Vincent van Gogh | FID: 71.82 -> 90.31 | Hurt |
| increase fine-tuning from 100 to 200 epochs (ablation) | E²GAN · Flickr Scenery concepts | FID: 115.32 -> 114.17; 110.87 -> 111.93; 122.77 -> 124.91 | No clear effect |
Surprising findings
- Pre-training pix2pix on multiple concepts gives little benefit and sometimes worsens FID, while the E$^2$GAN architecture benefits from it.
- Removing cross-attention during fine-tuning substantially degrades FID (e.g., 71.82 to 90.31 for Van Gogh style).
Limitations
- Diffusion models sometimes fail to produce good edits for diverse prompts, restricting dataset construction.
- Generating paired data with diffusion models remains expensive.
- Longer training did not improve results, and the cluster-number sensitivity means data selection is not uniformly beneficial.
Experimental setup
Datasets: FFHQ Flickr-Scenery AFHQ Facades Cityscapes Edges to Shoes Horse2Zebra
Benchmarks: FFHQ concept datasets Flickr-Scenery concept datasets AFHQ Facades Cityscapes Edges to Shoes Horse2Zebra
Metrics: Clean FID mIoU latency FLOPs parameter count training time
Baselines: pix2pix pix2pix-zero-distilled Co-Mod-GAN GAN Compression GAN Slimming CycleGAN CUT
Builds on: pix2pix pix2pix-zero LoRA Co-Mod-GAN Instruct-Pix2Pix
Closest papers by meaning
- Navigating the GAN Parameter Space for Semantic Image Editing (CVPR 2021 · similarity 0.91)
- EditGAN: High-Precision Semantic Image Editing (NeurIPS 2021 · similarity 0.90)
- Scaling Up GANs for Text-to-Image Synthesis (CVPR 2023 · similarity 0.90)
- Neural Photo Editing with Introspective Adversarial Networks (ICLR 2017 · similarity 0.90)
- Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (ICLR 2021 · similarity 0.90)
- Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.