Research map / Vision-language and multimodal / Human and embodied modeling
GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenVP, a hierarchical VAE-based generative RPM solver trained with ELBO plus cross-puzzle and cross-candidate masked contrastive losses and a mixture-of-experts rule predictor, which solves and generates Raven-style puzzles.
Problem
Generative abstract visual reasoning on Raven's Progressive Matrices: existing generative solvers are sensitive to distractors, struggle with large solution spaces, and only reconstruct missing panels rather than generating full puzzles from rules.
Approach
Models the entire RPM generation process with a graphical hierarchical VAE (image-level and puzzle-level latents), infers rules with a mixture of rule predictors over different context subsets, and adds global cross-puzzle and local cross-candidate masked contrastive losses on rule matrices.
Key results
- GenVP achieves 94.7/96.1% average RPM solving accuracy on RAVEN/I-RAVEN, versus 90.0/92.1% for RAISE.
- On OOD size extrapolation (size-e), GenVP reaches 45.0/65.5% average accuracy versus 28.5/43.6% for reproduced RAISE.
- GenVP generates complete RPMs with 78.0% average rule coherence on RAVEN-based layouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive training to ELBO training (ablation) | GenVP · RAVEN avg accuracy | accuracy (%): 50.5 -> 94.7 | Helped |
| add contrastive training to ELBO training (ablation) | GenVP · VAD dt accuracy | accuracy (%): 67.0 -> 95.8 | Helped |
| add contrastive training to ELBO training (ablation) | GenVP · PGM n accuracy | accuracy (%): 47.4 -> 79.5 | Helped |
| use equal-weight average instead of weighted average for MoE (ablation) | GenVP · RAVEN-FAIR 3×3 | accuracy (%): 89.2 -> 88.7 | Hurt |
| use argmax of the average rule predictions for MoE (ablation) | GenVP · RAVEN-FAIR 2×2 | accuracy (%): 96.1 -> 75.8 | Hurt |
| use product of rule predictions for MoE (ablation) | GenVP · RAVEN-FAIR 2×2 | accuracy (%): 96.1 -> 81.2 | Hurt |
| use argmax of the product of rule predictions for MoE (ablation) | GenVP · RAVEN-FAIR 2×2 | accuracy (%): 96.1 -> 68.2 | Hurt |
| use a learned neural-network MoE instead of weighted average (ablation) | GenVP · RAVEN-FAIR 3×3 | accuracy (%): 89.2 -> 90.2 | Helped |
| use a learned neural-network MoE instead of weighted average (ablation) | GenVP · PGM n accuracy | accuracy (%): 79.5 -> 85.5 | Helped |
| use a learned neural-network MoE instead of weighted average (ablation) | GenVP · PGM ar accuracy | accuracy (%): 17.0 -> 15.5 | Hurt |
| use a learned neural-network MoE instead of weighted average (ablation) | GenVP · VAD dt accuracy | accuracy (%): 95.8 -> 96.3 | Helped |
| use a learned neural-network MoE instead of weighted average (ablation) | GenVP · VAD e accuracy | accuracy (%): 81.4 -> 80.1 | Hurt |
Surprising findings
- Training with ELBO only (no contrastive terms) yields 50.5% average accuracy on RAVEN, far below the 94.7% with contrastive training.
- Color OOD performance drops sharply in o-ic and o-ig layouts because outer objects were always white in training and gray at test time.
Limitations
- Poor generalization to size extrapolation with very small objects, especially in o-ic, o-ig and 3x3 layouts.
- Struggles with complex configurations (o-ig, 2x2, 3x3) and intricate arithmetic rules.
- Low PGM neutral-like performance in extrapolation and some PGM splits (e.g., sc, ar) remain near chance-level.
Experimental setup
Datasets: RAVEN I-RAVEN RAVEN-FAIR PGM VAD
Benchmarks: RAVEN I-RAVEN RAVEN-FAIR PGM VAD
Metrics: puzzle-solving accuracy rule prediction accuracy generation coherence
Baselines: GCA ALANS PRAE RAISE
Builds on: RAISE GCA DAREN MLCL
Closest papers by meaning
- Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection (ICLR 2024 · similarity 0.88)
- Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning (ICML 2025 · similarity 0.87)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.87)
- GoT-R1: Unleashing Reasoning Capability of Autoregressive Visual Generation with Reinforcement Learning (ICLR 2026 · similarity 0.86)
- Hint2Gen: Bridging Understanding and Generation via Code-structured Hints (CVPR 2026 · similarity 0.86)
- PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.