Research map / Vision-language and multimodal / Human and embodied modeling
Learning to Infer Generative Template Programs for Visual Concepts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns neural inference networks that infer Template Programs (partial DSL programs) for groups of visual exemplars, supporting few-shot generation and co-segmentation across 2D layouts, Omniglot, and 3D shapes.
Problem
Inferring structured, programmatic representations of visual concepts from a small group of examples in a domain-general way, supporting tasks like few-shot generation and co-segmentation.
Approach
Represent a concept as a Template Program, a partially specified DSL program with HOLE tokens and parameter relations, inferred by autoregressive TemplateNet, ExpansionNet and ParamNet, trained on synthetic DSL samples and then bootstrapped with unsupervised self-training, latent-execution self-training and wake-sleep finetuning on concept groups.
Key results
- On 2D layouts few-shot generation, the method reaches FD 30.7 and Conf 90.9 versus 81.9 FD for VHE, outperforming task-specific alternatives.
- On Omniglot few-shot generation, the method achieves FD 115 and Conf 59.9, beating BPL (FD 130) and GNS (FD 123) on these metrics.
- In a perceptual study with 20 participants, generations were preferred over arVHE at 94% (Omniglot) and 84% (3D shapes), and over GNS at 64%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Template Programs instead of task-specific generation methods | few-shot generation · 2D Layouts | FD: 45.3 -> 30.7 | Helped |
| Use Template Programs instead of task-specific generation methods | few-shot generation · Omniglot | FD: 137 -> 115 | Helped |
| Use Template Programs instead of BAE | co-segmentation · 2D Layouts | mIoU: 34.5 -> 82.5 | Helped |
| Use Template Programs instead of BAE | co-segmentation · 3D Shapes | mIoU: 53.2 -> 68.6 | Helped |
| Use Template Programs instead of BPL | Omniglot co-segmentation · Omniglot | mIoU: 79.9 -> 78.7 | Hurt |
| Remove parameter relationships (ablation) | Ours · 2D Layouts few-shot generation | FD: 100% -> 78.5% | Hurt |
| Add parameter relationships for float-typed arguments (ablation) | Ours · 2D Layouts few-shot generation | FD: 100% -> 93.9% | Hurt |
| Remove HOLE tokens (ablation) | Ours · 2D Layouts few-shot generation | FD: 100% -> 96.3% | Hurt |
| Remove the Structural Expansion step (ablation) | Ours · 2D Layouts few-shot generation | FD: 100% -> 80.3% | Hurt |
| Remove bootstrapped finetuning (ablation) | Ours · 2D Layouts few-shot generation | FD: 100% -> 57.6% | Hurt |
| Remove parameter relationships (ablation) | Ours · 2D Layouts co-segmentation | mIoU: 100% -> 89.3% | Hurt |
| Remove HOLE tokens (ablation) | Ours · 2D Layouts co-segmentation | mIoU: 100% -> 97.4% | Hurt |
Surprising findings
- Domain-general method slightly outperforms domain-specific BPL and GNS on few-shot generation consistency and fidelity metrics on Omniglot, despite never training on human stroke data.
Limitations
- Only visual input groups of size 5 are considered in experiments.
- Template relations are limited to static assignment and variable reuse; continuous relations are not well modeled.
- Noisy or outlier input groups are not handled; the method may degenerate to a single HOLE token.
Experimental setup
Datasets: Omniglot PartNet synthetic 2D layout data sampled from DSL
Benchmarks: Omniglot 2D Layouts 3D Shapes (PartNet)
Metrics: Frechet Distance Confidence Minimum Matching Distance Coverage mIoU reconstruction IoU Chamfer distance
Baselines: BPL GNS FSDM VHE arVHE BAE-NET
Builds on: PLAD BPL ShapeAssembly DreamCoder BAE-NET
Closest papers by meaning
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.88)
- GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules (ICLR 2024 · similarity 0.87)
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.87)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.86)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.86)
- Compositional Scene Understanding through Inverse Generative Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.