Research map / Architectures and efficiency / RNNs and normalization
Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes training dynamics of a single-layer GAN with a multi-feature (orthonormal, same-dimension) discriminator via scaling-limit ODE/SDE analysis, compares it to Oja's method and GROUSE for subspace learning, and validates on synthetic data, MNIST and Olivetti Faces.
Problem
Understanding training dynamics of single-layer GANs for online subspace learning in high dimensions, particularly when the discriminator learns all features at once rather than sequentially.
Approach
Replaces the single-feature discriminator with a multi-feature linear discriminator (W in R^{n x d}, orthonormal) so all generator features are learned jointly; derives macroscopic ODEs and microscopic SDE via a scaling limit, and extends analysis to unknown feature dimensionality via a Grassmannian uplifting trick.
Key results
- On MNIST, the multi-feature GAN after 1 epoch reaches Grassmann distance 2.46 versus 3.17 for the single-feature discriminator after 5 epochs.
- ODE predictions for macroscopic states match simulations across noise levels; the multi-feature discriminator outperforms the single-feature one in off-diagonal initialization, with the gap growing as initialization approaches 0.
- On Olivetti Faces and MNIST, the GAN learns more interpretable features than Oja's method, though Oja learns faster initially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a multi-feature instead of a single-feature discriminator | Single-layer GAN · MNIST, 16 features | Grassmann distance: 2.46 vs 3.17 | Helped |
| Use a multi-feature instead of a single-feature discriminator | Single-layer GAN · Olivetti Faces | Grassmann distance | Helped |
| Use a multi-feature instead of a single-feature discriminator | Single-layer GAN · MNIST, 36 features | learned basis quality | Helped |
| Use a multi-feature instead of a single-feature discriminator | Single-layer GAN · Synthetic subspace learning, off-diagonal initialization | steady-state cosine similarity | Helped |
| Initialize macroscopic states with off-diagonal entries | Multi-feature GAN · Synthetic subspace learning | steady-state cosine similarity | Helped |
| Increase noise to eta = 5 or higher | Multi-feature GAN · Synthetic subspace learning | learning | Hurt |
| Train the sequential discriminator beyond 5 epochs | Single-feature GAN · MNIST | Grassmann distance | No clear effect |
| Use GAN-learned features instead of Oja-learned features | Subspace learning on Olivetti Faces · Olivetti Faces | feature quality | Helped |
| Use GAN-learned features instead of Oja-learned features | Subspace learning on MNIST · MNIST, 16 features | feature quality | Helped |
Surprising findings
- Oja's method learns faster than the GAN early in training on Olivetti Faces, though the GAN eventually surpasses it.
- A powerful (same-dimension) discriminator can yield faster training and better performance than a weak discriminator when learning rates are chosen carefully.
Limitations
- Analysis restricted to a single-layer linear generator and discriminator with orthonormal discriminator weights (A.6), and to a limiting regime n to infinity.
- Off-diagonal steady-state fixed points lack closed-form characterization.
- Real-data experiments use hand-tuned hyperparameters and an approximation of the true subspace via PCA.
Experimental setup
Datasets: MNIST Olivetti Faces synthetic spiked covariance data
Benchmarks: MNIST Olivetti Faces synthetic spiked covariance
Metrics: Grassmann distance cosine similarity of macroscopic states
Baselines: Oja's method GROUSE single-feature (sequential) GAN discriminator
Builds on: Oja's method GROUSE single-feature GAN training dynamics analysis (Wang et al.-style ODE analysis)
Closest papers by meaning
- On the Limitations of First-Order Approximation in GAN Dynamics (ICML 2018 · similarity 0.88)
- Ensembling Off-the-Shelf Models for GAN Training (CVPR 2022 · similarity 0.87)
- Low-Rank Subspaces in GANs (NeurIPS 2021 · similarity 0.87)
- Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator (ICML 2025 · similarity 0.87)
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.86)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.