Research map / Label-efficient and robust learning / Recognition and long-tail
Learning Group Actions on Latent Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns group actions (rotations, contrast affine maps, cyclic groups) directly on the latent space of an autoencoder with U-Net-style skip connections and attention, so that rendering transformed images works even when the group acts only on latent factors such as occluded or foreground content.
Problem
Existing group-action models for autoencoders act on the data space, which cannot capture cases where the group acts only on latent factors (e.g., occluded digits, foreground objects on fixed backgrounds, or 3D rotations seen as 2D images).
Approach
Apply the group action to a split latent representation (varying part acted on by the group, invariant part fixed) and decode the transformed latent with a flexible encoder/decoder without group-specific layers; attention-based skip connections help mix spatial information. Theory shows that decoders invertible on the left by the encoder induce data-space group actions, so the method generalizes d
Key results
- On rotated and blocked MNIST, the model reaches 23.55 PSNR and 1.000 SSIM versus 14.05 PSNR for Winter et al.
- On NMR, the model reaches 28.91 PSNR, 0.947 SSIM, and 0.050 LPIPS, beating Sajjadi et al. (27.87 PSNR) and Dupont et al. (26.91 PSNR).
- On plane in the sky, the model reaches 25.24 PSNR and 0.821 SSIM versus 24.25 PSNR for Dupont et al. and 23.53 for Sajjadi et al.; ~0.33M parameters on MNIST datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add skip connections and LPIPS loss (ablation) | Ours · NMR | PSNR: 27.31 -> 28.91 | Helped |
| add skip connections and LPIPS loss (ablation) | Ours · NMR | SSIM: 0.933 -> 0.947 | Helped |
| add skip connections and LPIPS loss (ablation) | Ours · NMR | LPIPS: 0.080 -> 0.050 | Helped |
| add skip connections (ablation) | Ours · NMR | PSNR: 26.96 -> 28.91 | Helped |
| add skip connections (ablation) | Ours · NMR | SSIM: 0.930 -> 0.947 | Helped |
| add skip connections (ablation) | Ours · NMR | LPIPS: 0.063 -> 0.050 | Helped |
| add LPIPS loss (ablation) | Ours · NMR | PSNR: 28.09 -> 28.91 | Helped |
| add LPIPS loss (ablation) | Ours · NMR | SSIM: 0.941 -> 0.947 | Helped |
| add LPIPS loss (ablation) | Ours · NMR | LPIPS: 0.072 -> 0.050 | Helped |
| add the encoder-decoder consistency loss (ablation) | Ours on Rotated MNIST · Rotated MNIST test set | reconstruction performance | No clear effect |
Surprising findings
- Hwang et al.'s normalizing-flow model with Euclidean-group assumption performs poorly on rotations, and its diffeomorphism constraint hurts on the blocked MNIST.
Limitations
- Requires ground-truth group actions during training; semi-supervised or unsupervised extension left for future work
- Tested on five image datasets and a limited set of groups (SO(2), SO(3), Aff(1), cyclic)
- No error bars or statistical significance reported
Experimental setup
Datasets: MNIST Human Connectome Project brain MRI ShapeNet SWIMSEG sky dataset
Benchmarks: Rotated MNIST Rotated and blocked MNIST Brain MRI NMR Plane in the sky
Metrics: PSNR SSIM LPIPS
Baselines: Hwang et al. (2021) VAE group action model Winter et al. (2022) equivariant autoencoder Dupont et al. (2020) equivariant novel view synthesis Sajjadi et al. (2022) Scene Representation Transformer
Builds on: U-Net Transformer attention LPIPS
Closest papers by meaning
- Homomorphism AutoEncoder --- Learning Group Structured Representations from Observed Transitions (ICML 2023 · similarity 0.90)
- Unsupervised Learning of Group Invariant and Equivariant Representations (NeurIPS 2022 · similarity 0.87)
- Learning from Synthetic Human Group Activities (CVPR 2024 · similarity 0.86)
- Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions (NeurIPS 2025 · similarity 0.86)
- World Models as Group Actions (NeurIPS 2026 · similarity 0.85)
- Variational Partial Group Convolutions for Input-Aware Partial Equivariance of Rotations and Color-Shifts (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.