Research map / Label-efficient and robust learning / Knowledge distillation

One Unified Representation: Resolving the Appearance-Semantics Dilemma via Structural Regularization

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Patient consistent distillation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a structurally regularized unified visual representation that adds appearance detail to a semantic ViT encoder for both reconstruction and DiT-based generation, avoiding the semantic collapse seen when fully fine-tuning with reconstruction loss.

Problem

Representation Autoencoders (RAE) use frozen semantic ViT features that lack appearance details, giving suboptimal reconstruction; naively fine-tuning the encoder with reconstruction loss causes semantic collapse, where the encoder overfits to appearance and loses semantic discriminability.

Approach

Instead of full fine-tuning, regularize the encoder structurally so that it gains appearance detail while preserving semantic structure, yielding one representation usable for both reconstruction and generation (details not available beyond the abstract).

Surprising findings

• Full fine-tuning of the semantic encoder with reconstruction loss causes catastrophic semantic collapse, with the encoder overfitting to appearance and losing semantic discriminability.

Experimental setup

Baselines: Representation Autoencoder (RAE)

Builds on: Representation Autoencoder (RAE) DiT

Closest papers by meaning

• Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing (ICML 2026 · similarity 0.88)
• Unified Latent Space for Understanding and Generation via Semantic Auto-encoder (CVPR 2026 · similarity 0.88)
• IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder (NeurIPS 2026 · similarity 0.88)
• Latent Harmony: Synergistic Unified UHD Image Restoration via Latent Space Regularization and Controllable Refinement (NeurIPS 2025 · similarity 0.87)
• Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.86)
• Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.86)