Research map / Label-efficient and robust learning / Recognition and long-tail
How Learning by Reconstruction Produces Uninformative Features For Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that reconstruction-based representation learning allocates capacity to high-variance pixel subspaces that are uninformative for perception, and analyzes how denoising noise choices (masking vs. Gaussian) affect alignment.
Problem
Reconstruction-based representation learning (autoencoders, denoising and masked autoencoders) yields interpretable reconstructions but representations that are poorly suited for perception tasks and require long training and fine-tuning.
Approach
Derives a closed-form linear alignment measure between reconstruction and supervised objectives, showing the top-variance subspace favored by reconstruction is least informative for classification; shows masking-based denoising provably changes the alignment while additive Gaussian noise does not.
Key results
- On TinyImageNet, supervised accuracy with images projected onto the top subspace (90% of pixel variance) is 45% versus 55% using the bottom subspace (20% of variance).
- Adding a supervised head to a ResNet34 autoencoder yields much higher classification accuracy at identical reconstruction train/test loss on Imagenette.
- Theory shows additive isotropic Gaussian noise leaves the optimal linear supervised solution unchanged for any noise level, while masking can change alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the bottom pixel-variance subspace instead of the top subspace | ResNet9 trained and validated on projected images · TinyImageNet | top-1 accuracy: 45% -> 55% | Helped |
| Fine-tune the learned representation | MAE · ImageNet top-1 classification | top-1 accuracy: 70% -> 84% | Helped |
| Add masking during reconstruction learning | MAE · ImageNet top-1 | top-1 accuracy: about 50% -> 74% | Helped |
| Use additive Gaussian noise instead of no denoising noise (ablation) | Linear denoising autoencoder representation · Supervised tasks | supervised task performance | No clear effect |
| Use masking noise rather than no denoising noise (ablation) | Denoising autoencoder representation · CIFAR100 | alignment between reconstruction and perception tasks | Helped |
| Use masking noise rather than no denoising noise (ablation) | Denoising autoencoder representation · ArabicDigits | alignment between reconstruction and perception tasks | Hurt |
| Vary the masking patch shape, ratio, and dataset (ablation) | Denoising autoencoder representation · Multiple datasets | alignment between reconstruction and perception tasks | Mixed |
| Add a supervised classification loss alongside reconstruction loss | ResNet34 autoencoder · Imagenette (Imagenet-10) | classification performance | Helped |
| Add a supervised classification loss alongside reconstruction loss | ResNet34 autoencoder · Imagenette (Imagenet-10) | reconstruction loss | No clear effect |
| Keep the bottom subspace instead of the top subspace | Deep networks trained and tested on filtered images · Multiple datasets | test accuracy | Helped |
Surprising findings
- Keeping the low-variance bottom subspace of pixels yields higher supervised accuracy than keeping the high-variance top subspace.
- Additive Gaussian denoising provably has no effect on the optimal linear supervised performance regardless of noise level.
Limitations
- Theoretical analysis is in the linear regime; extension to nonlinear networks holds only at initialization or with linear decoders.
- Best masking shape and ratio vary by dataset, so noise strategy requires per-dataset tuning without knowledge of the perception task.
Experimental setup
Datasets: TinyImageNet ImageNet ImageNette CIFAR10 CIFAR100 ArabicDigits
Benchmarks: TinyImageNet ImageNet ImageNette CIFAR10 CIFAR100 ArabicDigits
Metrics: top-1 accuracy alignment(k) reconstruction loss
Baselines: Masked Autoencoders PCA
Builds on: Masked Autoencoders Denoising Autoencoders Principal Component Analysis
Closest papers by meaning
- Rethinking Reconstruction and Denoising in the Dark: New Perspective, General Architecture and Beyond (CVPR 2025 · similarity 0.86)
- Traversing Distortion-Perception Tradeoff using a Single Score-Based Generative Model (CVPR 2025 · similarity 0.85)
- Your contrastive learning problem is secretly a distribution alignment problem (NeurIPS 2024 · similarity 0.85)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.85)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.84)
- Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.