Research map / Image generation / Autoregressive and latent generation
Interpretable Generative Models through Post-hoc Concept Bottlenecks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes post-hoc concept-bottleneck autoencoder (CB-AE) and concept controller (CC) that make frozen pretrained GANs and diffusion models interpretable and steerable, trained with pseudo-labels instead of real concept-labeled data.
Problem
Building interpretable generative models with concept bottlenecks is expensive: prior CBGM requires training generators from scratch with concept-labeled real images.
Approach
Insert a concept-bottleneck autoencoder into the intermediate latent of a frozen pretrained generator, trained with reconstruction, concept-alignment, and intervention losses using pseudo-labels from a supervised or CLIP-based source; a lighter concept controller only predicts concepts for optimization-based steering.
Key results
- CB-AE-GAN+opt-int reaches 59.9% steerability on CelebA (8 concepts) vs. 25.6% for CBGM, with CC+opt-int achieving 51.14% vs. 25.60% on CelebA GAN (5k samples).
- CC trains in 6 V100-hours vs. 50 for CBGM on CelebA-HQ StyleGAN2 (8.3× faster), and CB-AE trains in 14 hours (3.5× faster), with FID 7.65 for CC+opt-int matching base model 7.66.
- Human study agreement with automated concept accuracy and steerability is similar (e.g., CB-AE smiling concept accuracy 92.38% automated vs. 86.35% human).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add optimization-based intervention to CB-AE | CB-AE-GAN · CelebA (64×64), 8 concepts | steerability: 47.1 ± 0.5 -> 59.9 ± 1.8 | Helped |
| use CC with optimization-based intervention instead of CB-AE with optimization-based intervention | GAN generative model · CelebA (64×64), 8 concepts | steerability: 59.9 ± 1.8 vs. 50.8 ± 1.2 | Mixed |
| add optimization-based intervention to CB-AE | CB-AE-DDPM · CelebA (64×64), 8 concepts | steerability: 23.1 ± 0.9 -> 37.5 ± 1.3 | Helped |
| use optimization-based intervention with CC instead of CB-AE | DDPM generative model · CelebA (64×64), 8 concepts | steerability: 37.5 ± 1.3 -> 41.8 ± 1.8 | Helped |
| use CB-AE instead of CBGM | CelebA-HQ StyleGAN2 · CelebA-HQ | FID: 9.1 -> 9.52 | Hurt |
| use CC with optimization-based intervention instead of CBGM | CelebA-HQ StyleGAN2 · CelebA-HQ | FID: 9.1 -> 7.65 | Helped |
| train post-hoc CB-AE instead of CBGM from scratch | CelebA-HQ StyleGAN2 · CelebA-HQ | training time: 50 -> 14 | Helped |
| train post-hoc CC instead of CBGM from scratch | CelebA-HQ StyleGAN2 · CelebA-HQ | training time: 50 -> 6 | Helped |
| add image reconstruction loss Lr2 (ablation) | CB-AE · CelebA-HQ StyleGAN2, CLIP-zero-shot pseudo-labels | FID: 11.06 -> 9.88 | Helped |
| add intervened concept loss Li1 (ablation) | CB-AE with Lr2 · CelebA-HQ StyleGAN2, CLIP-zero-shot pseudo-labels | steerability: 6.51 -> 9.98 | Helped |
| add cyclic intervened concept loss Li2 (ablation) | CB-AE with Lr2 · CelebA-HQ StyleGAN2, CLIP-zero-shot pseudo-labels | steerability: 6.51 -> 13.03 | Helped |
| add both intervention losses Li1 and Li2 (ablation) | CB-AE with Lr2 · CelebA-HQ StyleGAN2, CLIP-zero-shot pseudo-labels | steerability: 9.98 -> 20.61; 13.03 -> 20.61 | Helped |
Surprising findings
- CB-AE outperforms CC on concept accuracy only for StyleGAN2, which the authors attribute to multi-objective training being easier with clean GAN latents than noisy DDPM latents.
- CB-AE with CB intervention gives worse FID than CBGM (9.52 vs. 9.1), showing a quality–interpretability tradeoff.
Limitations
- Image quality drops relative to CBGM for the CB-AE model (FID 9.52 vs. 9.1).
- CBGM results could not be reproduced, so baseline numbers are taken from the paper for some settings.
- Concept accuracy and steerability depend on pseudo-label quality; CLIP zero-shot pseudo-labels yield notably lower steerability.
Experimental setup
Datasets: CelebA CelebA-HQ CUB
Benchmarks: CelebA CelebA-HQ CUB
Metrics: concept accuracy steerability FID training time
Baselines: CBGM CGAN ACGAN CF-DDPM CG-DDPM CB-DDPM
Builds on: CBGM concept bottleneck models I-RFGSM CLIP
Closest papers by meaning
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.90)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.89)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.89)
- Concept-Based Unsupervised Domain Adaptation (ICML 2025 · similarity 0.88)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.88)
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.