Research map / Detection and segmentation / Open-set and few-shot detection
Learning from Pattern Completion: Self-supervised Controllable Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-supervised controllable generation (SCG) framework that uses a modular autoencoder trained with equivariance constraints to form functional modules, which then condition a ControlNet-style adapter on SD1.5 via self-supervised pattern completion.
Problem
Controllable generation methods such as ControlNet need annotated paired conditions (edges, depth, segmentation, pose), which limits scalability; the paper asks whether brain-inspired modularity and pattern completion can yield self-supervised controllable generation.
Approach
Train a modular autoencoder with translation/rotation equivariance and symmetry losses so latent modules specialize (color, brightness, edges) without labels, then use subsets of these modules as conditions for a ControlNet adapter that learns to complete the full image, enabling zero-shot associative generation from sketches, paintings and graffiti.
Key results
- On MS-COCO validation, SCG conditioned on learned modules achieves higher PSNR and SSIM than ControlNet with Canny edges, with HC0 best on FID and CLIP similarity.
- In subjective evaluation on oil paintings, SCG has significantly higher winning rates than ControlNet in both fidelity and aesthetics.
- On ancient graffiti, SCG shows similar fidelity winning rate and a significantly higher aesthetics winning rate than Canny-based ControlNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCG with autonomously differentiated modules instead of ControlNet with Canny edges | SCG vs. ControlNet · MS-COCO validation | PSNR | Helped |
| Use SCG with autonomously differentiated modules instead of ControlNet with Canny edges | SCG vs. ControlNet · MS-COCO validation | SSIM | Helped |
| Use SCG conditioned by HC0–HC3 instead of ControlNet conditioned by Canny edges | SCG vs. ControlNet · MS-COCO validation | FID | Helped |
| Use SCG conditioned by HC0–HC3 instead of ControlNet conditioned by Canny edges | SCG vs. ControlNet · MS-COCO validation | CLIP similarity | Helped |
| Use SCG with HC1 instead of ControlNet with Canny edges | SCG vs. ControlNet · Oil painting associative generation | aesthetics winning rate | Helped |
| Use SCG with HC1 instead of ControlNet with Canny edges | SCG vs. ControlNet · Oil painting associative generation | fidelity winning rate | Helped |
| Use SCG with HC3 instead of ControlNet with Canny edges | SCG vs. ControlNet · Ancient graffiti associative generation | aesthetics winning rate | Helped |
| Use SCG with HC3 instead of ControlNet with Canny edges | SCG vs. ControlNet · Ancient graffiti associative generation | fidelity winning rate | No clear effect |
| Add equivariance constraints to the modular autoencoder (ablation) | Modular Autoencoder · ImageNet feature specialization | functional specialization | Helped |
| Remove the symmetric loss (ablation) | Modular Autoencoder · Feature specialization | within-module feature correlation | Hurt |
| Vary the number of modules (ablation) | Modular Autoencoder · Feature differentiation | functional specialization | No clear effect |
| Remove translation transforms from training data (ablation) | Modular Autoencoder · Feature specialization | direction selectivity | Hurt |
Surprising findings
- Learned modules, without any labels, separate into color, brightness, and edge-frequency functional groups, with brain-like orientation selectivity and center-surround receptive fields.
Limitations
- The equivariance constraint is a simplified, preliminary version that yields only low-level feature specialization (no depth, motion, or semantic contours).
- Fully self-supervised training falls short of dedicated supervised methods on specific tasks.
- Subjective evaluations used small sets (six oil painting images, 37-40 participants), and statistical significance is not reported.
Experimental setup
Datasets: MNIST ImageNet (100 classes) MS-COCO 2017 train MS-COCO validation
Benchmarks: MS-COCO validation set zero-shot sketches, oil paintings, wash and ink paintings, ancient graffiti
Metrics: PSNR SSIM FID CLIP similarity subjective winning rate on fidelity and aesthetics
Baselines: ControlNet (Canny edge condition) ControlNet (depth, normal, segmentation conditions)
Builds on: ControlNet Stable Diffusion 1.5 group convolution capsule networks
Closest papers by meaning
- Self-Supervised Learning Through Efference Copies (NeurIPS 2022 · similarity 0.85)
- Representing Part-Whole Hierarchy with Nested Neuronal Coherence (NeurIPS 2026 · similarity 0.85)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.85)
- MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning (AAAI 2025 · similarity 0.84)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.84)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.