Research map / Image generation / GANs

SMaRt: Improving GANs with Score Matching Regularity

ICML 2024 · Image generation › GANs · approach: Stabilized GAN training · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMaRt, a plug-in regularizer that adds a score-matching loss from a pre-trained diffusion model to GAN generator training to push off-manifold samples back onto the data manifold, improving StyleGAN2, BigGAN and Aurora FID.

Problem

GAN generator loss gives no gradient when generated samples lie off the real data manifold on a positive-measure set, causing gradient vanishing and poor synthesis quality on diverse data.

Approach

Add a score-matching regularity term, computed with a pre-trained diffusion model's noise predictor via finitely many one-step DDIM-style refinements, to the GAN generator objective so that off-manifold generated samples keep receiving gradient toward the data manifold; uses lazy regularization and a narrowed timestep interval to limit cost.

Key results

• Aurora on ImageNet 64x64 improves FID from 8.87 to 7.11 with SMaRt (7.16 ± 0.05 with 32 GPUs in multi-run table).
• StyleGAN2 + SMaRt reaches FID 2.06 on CIFAR10 (IS 10.22) and 1.98 on LSUN Bedroom 256x256, versus 2.35 for the StyleGAN2 baseline on LSUN Bedroom.
• BigGAN + SMaRt improves FID on ImageNet 128x128 from 10.76 to 9.49.

Measured effects reported in the paper

Surprising findings

• Vanilla GAN and DiffusionGAN produce off-grid samples on a discrete 49-Gaussian toy problem, while SMaRt recovers the discrete distribution.

Limitations

• Performance depends heavily on hyper-parameters (lambda_score, timestep interval, regularization frequency) and their optimality is unexplored.
• Training cost still increases because the pre-trained diffusion model must be evaluated, slowing training.
• Still does not outperform diffusion models in overall quality.

Experimental setup

Datasets: CIFAR10 ImageNet 64x64 ImageNet 128x128 LSUN Bedroom 256x256 synthetic 49-Gaussian toy mixture

Benchmarks: CIFAR10 ImageNet 64x64 ImageNet 128x128 LSUN Bedroom 256x256

Metrics: FID Inception Score Precision Recall

Baselines: StyleGAN2 BigGAN Aurora DiffusionGAN Consistency Model StyleGAN2-ADA

Builds on: GAN DDIM StyleGAN2 BigGAN Aurora

Closest papers by meaning

• Improving GAN with Neighbors Embedding and Gradient Matching (AAAI 2019 · similarity 0.91)
• Geometry Score: A Method For Comparing Generative Adversarial Networks (ICML 2018 · similarity 0.90)
• A Large-Scale Study on Regularization and Normalization in GANs (ICML 2019 · similarity 0.90)
• The GAN is dead; long live the GAN! A Modern GAN Baseline (NeurIPS 2024 · similarity 0.89)
• MCGAN: Enhancing GAN Training with Regression-Based Generator Loss (AAAI 2025 · similarity 0.88)
• Adversarial Score Distillation: When score distillation meets GAN (CVPR 2024 · similarity 0.88)