Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning to Edit Visual Programs with Self-Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a local program-edit network for visual program induction, jointly finetuned with a one-shot model via self-supervised bootstrapping, and uses edit-based population search at inference.
Problem
Visual program induction in domains without program annotations; one-shot autoregressive program predictors cannot revise complete programs based on execution feedback.
Approach
An edit network consumes a full input program, its execution, and a visual target, and predicts a local edit operation (type, location, parameters); it is trained self-supervisedly by findEdits on start/end program pairs sampled within a PLAD-style bootstrapped finetuning loop, and combined with the one-shot model in an evolutionary population search.
Key results
- On Layout, OS+Edit reaches 0.98 cIoU vs 0.94 for OS Only; on 2D CSG Chamfer distance 0.111 vs 0.156; on 3D CSG IoU 85.3 vs 83.3.
- With only 100 training shapes, the joint paradigm matches OS Only trained with 10x more data on 2D CSG.
- On out-of-distribution challenge tasks, OS+Edit improves Layout cIoU from 75.8 to 87.6 and 3D CSG IoU from 60.8 to 70.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a jointly trained edit network and use it with the one-shot model | OS Only · Layout test set | cIoU: 0.94 -> 0.98 | Helped |
| add a jointly trained edit network and use it with the one-shot model | OS Only · 2D CSG test set | CD: 0.156 -> 0.111 | Helped |
| add a jointly trained edit network and use it with the one-shot model | OS Only · 3D CSG test set | IoU: 83.3 -> 85.3 | Helped |
| add a jointly trained edit network and use it with the one-shot model | OS Only · Layout challenge tasks | cIoU: 75.8 -> 87.6 | Helped |
| add a jointly trained edit network and use it with the one-shot model | OS Only · 3D CSG challenge tasks | IoU: 60.8 -> 70.9 | Helped |
| finetune both networks on the target data (ablation) | OS + Edit · 2D CSG test set | CD: 0.321 -> 0.111 | Helped |
| finetune the one-shot network (ablation) | OS + Edit · 2D CSG test set | CD: 0.230 -> 0.111 | Helped |
| finetune the edit network (ablation) | OS + Edit · 2D CSG test set | CD: 0.123 -> 0.111 | Helped |
| pretrain the edit network on synthetic data (ablation) | OS + Edit · 2D CSG test set | CD: 0.145 -> 0.111 | Helped |
| use local edit operations instead of predicting the next or final full program (ablation) | OS + Edit · Layout | reconstruction performance | Helped |
| source edit training pairs with findEdits rather than program corruption (ablation) | OS + Edit · Layout | reconstruction performance | Helped |
| resample the population according to reconstruction metric instead of applying the highest-likelihood edit (ablation) | OS + Edit inference · Layout | cIoU: 0.022 improvement from the first to the last round | Helped |
Surprising findings
- A sampling-based one-shot inference baseline (OS Only) is surprisingly strong when matched on inference time.
- GPT-4V zero-shot and in-context prompting performed worse than the proposed method on layout program induction.
Limitations
- Requires a domain-aware findEdits operation to derive edit supervision from start/end program pairs.
- Training requires an additional network, increasing pretraining and finetuning time.
- Benefits depend on more expensive inference search budgets.
Experimental setup
Datasets: CSGNet 2D CSG shapes PLAD 3D CSG shapes (ShapeNet) Layout scenes from prior work synthetic programs sampled from DSL grammars
Benchmarks: Layout test set (144 scenes) 2D CSG test set (3000 shapes) 3D CSG test set (1000 shapes) challenge splits (Layout, 3D CSG)
Metrics: cIoU Chamfer distance IoU
Baselines: OS Only UCSG-Net SIRI ROAP GPT-4V
Builds on: PLAD SIRI CSGNet
Closest papers by meaning
- Meta-CoT: Enhancing Granularity and Generalization in Image Editing (CVPR 2026 · similarity 0.87)
- Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement (CVPR 2024 · similarity 0.86)
- PairEdit: Learning Semantic Variations for Exemplar-based Image Editing (NeurIPS 2025 · similarity 0.85)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.85)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.85)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.