Research map / Image generation / Editing and style transfer
Rethinking the Paradigm of Content Constraints in Unpaired Image-to-Image Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnCo, which enforces content preservation in unpaired I2I by an MSE loss between same-stage encoder and decoder patch features of the generator, plus a parameter-free discriminator-guided patch sampling (DAG).
Problem
Unpaired image-to-image translation with GANs lacks sufficient content constraints and tends to model collapse; reconstruction-based methods are too strict and Siamese-network methods are costly and need a good feature extractor.
Approach
Constrain content inside the generator by matching representations of same-stage encoder and decoder patch features via a stop-gradient, negative-free MSE loss with projection and prediction heads, and choose patches with a discriminator-guided attention sampling rule (DAG) instead of random sampling.
Key results
- EnCo reaches FID 45.4 on Cityscapes, 54.7 on Cat->Dog and 38.7 on Horse->Zebra, the best FID among compared methods (second best 49.4, 60.7, 41.1).
- EnCo trains at 0.14 sec/iter, about 2.9x faster than CycleGAN and 1.7x faster than CUT (0.24).
- Replacing CUT's PatchNCE with EnCo's MSE loss lowers CUT's Cat->Dog FID from 76.2 to 63.8 and Horse->Zebra from 45.5 to 43.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EnCo instead of the strongest reported baseline | Unpaired image translation · Cat→Dog | FID: 60.7 -> 54.7 | Helped |
| Use EnCo instead of the strongest reported baseline | Unpaired image translation · Cityscapes | FID: 49.4 -> 45.4 | Helped |
| Use EnCo instead of QS-Attn | Unpaired image translation · Horse→Zebra | FID: 41.1 -> 38.7 | Helped |
| Use EnCo instead of QS-Attn | Unpaired image translation · Cityscapes | pixAcc: 79.9 -> 77.3 | Hurt |
| Add a second same-stage feature pair (h13,h18) (ablation) | EnCo · Cat→Dog | FID: 63.8 -> 63.8 | No clear effect |
| Add a second same-stage feature pair (h13,h18) (ablation) | EnCo · Horse→Zebra | FID: 43.1 -> 43.1 | No clear effect |
| Replace symmetric feature pairing with asymmetric pairs and alignment (ablation) | EnCo · Cat→Dog | FID: 54.7 -> 55.9 | Hurt |
| Replace DAG with uniform random sampling (ablation) | EnCo · Cat→Dog | FID: 54.7 -> 63.8 | Hurt |
| Replace DAG with uniform random sampling (ablation) | EnCo · Horse→Zebra | FID: 38.7 -> 43.1 | Hurt |
| Remove the predictor (ablation) | EnCo · Cat→Dog | FID: 54.7 -> 64.2 | Hurt |
| Remove layer normalization (ablation) | EnCo · Horse→Zebra | FID: 38.7 -> 46.9 | Hurt |
| Stop gradient on the encoder feature branch (ablation) | EnCo · Unpaired image translation | training stability | Helped |
Surprising findings
- Negative samples are not necessary for the encoder-decoder content constraint; a plain MSE loss with stop-gradient works instead of contrastive loss.
- Removing the stop-gradient causes training collapse on EnCo.
Limitations
- Symmetric encoder-decoder assumption; asymmetric variant (EnCo-dagger) shows slight degradation.
- Segmentation metrics on Cityscapes: pixAcc is slightly below QS-Attn.
- Identity loss used only in the first half of training because it hurts generative performance.
Experimental setup
Datasets: Cityscapes AFHQ Cat->Dog Horse->Zebra (ImageNet)
Benchmarks: Cityscapes Cat->Dog Horse->Zebra
Metrics: FID mAP pixAcc classAcc sec/iter memory (GB)
Baselines: CycleGAN MUNIT GcGAN CUT DCLGAN FSeSim MoNCE QS-Attn
Builds on: CUT CycleGAN SimSiam BYOL LSGAN
Closest papers by meaning
- TraVeLGAN: Image-To-Image Translation by Transformation Vector Learning (CVPR 2019 · similarity 0.90)
- ReMix: Towards Image-to-Image Translation With Limited Data (CVPR 2021 · similarity 0.90)
- Unsupervised Image-to-Image Translation With Generative Prior (CVPR 2022 · similarity 0.90)
- DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs (NeurIPS 2020 · similarity 0.89)
- E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.87)
- Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.