Research map / Video / Video generation
Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JPDVT, a conditional diffusion transformer that generates positional encodings for shuffled image patches or video frames conditioned on their visual content, solving image and video jigsaw puzzles including those with missing pieces.
Problem
Solving image and video jigsaw puzzles with large numbers of pieces and missing pieces, where discriminative position/permutation classifiers struggle.
Approach
Treat the puzzle as a set of (visual embedding, positional encoding) pairs and learn the joint distribution of positional encodings conditioned on visual content via a conditional DDPM with a ViT/video U-Net denoiser, so missing pieces can be handled by also generating their content.
Key results
- On 3x3 ImageNet-1k puzzles without missing pieces, JPDVT reaches 68.7% puzzle-level accuracy vs 48% for Deepzzle, and 83.3% piece-level.
- On JPwLEG-3, JPDVT achieves 71.3% piece-level accuracy vs 59.7% for Siamese (AAAI-2023).
- On 150-piece puzzles with 7% erosion, puzzle-level accuracy is 45% vs 10% for SJP-ED.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use JPDVT instead of Deepzzle | 3×3 image puzzle solving · ImageNet-1k val | puzzle-level accuracy: 48% -> 68.7% | Helped |
| use JPDVT instead of Deepzzle | 3×3 image puzzle solving · ImageNet-1k val | piece-level accuracy: 78% -> 83.3% | Helped |
| use JPDVT instead of Siamese | 3×3 image puzzle solving · JPwLEG-3 | piece-level accuracy: 59.7% -> 71.3% | Helped |
| use JPDVT instead of SJP-ED | 150-piece puzzle solving with 7% erosion · 150-piece puzzle | puzzle-level accuracy: 10% -> 45% | Helped |
| use JPDVT instead of STP | temporal puzzle solving · MovingMNIST, 20 pieces | normalized Kendall distance × 10³: 1.4 -> 0.7 | Helped |
| use JPDVT instead of J-VAD | temporal puzzle solving · CLEVRER, 32 pieces | normalized Kendall distance × 10³: 268.2 -> 2.6 | Helped |
| use JPDVT instead of J-VAD | temporal puzzle solving · UCF101, 16 pieces | normalized Kendall distance × 10³: 164.3 -> 4.2 | Helped |
| increase missing fragments from 0% to 11.1% (ablation) | JPDVT · ImageNet-1k val, 3×3 puzzle with gap | puzzle-level accuracy: 68.7% -> 41.5% | Hurt |
| increase missing fragments from 0% to 22.2% (ablation) | JPDVT · ImageNet-1k val, 3×3 puzzle with gap | puzzle-level accuracy: 68.7% -> 21.4% | Hurt |
| increase missing fragments from 0% to 33.3% (ablation) | JPDVT · ImageNet-1k val, 3×3 puzzle with gap | puzzle-level accuracy: 68.7% -> 14.9 % | Hurt |
| remove eroded gaps (ablation) | JPDVT · ImageNet-1k val, 3×3 puzzle with 0% missing | puzzle-level accuracy: 68.7% -> 99.4% | Helped |
Surprising findings
- STP obtains best results on QST, which the authors attribute to its use of PCA and ground-truth correspondences rather than a fair comparison.
Limitations
- Performance decreases as the percentage of missing pieces increases, especially with eroded gaps (e.g., 14.9% puzzle-level at 33% missing with gaps).
- Temporal inference is about an order of magnitude slower than VCOP and J-VAD.
Experimental setup
Datasets: ImageNet-1k JPwLEG-3 MovingMNIST CLEVRER QST UCF101
Benchmarks: ImageNet-1k val (3x3 puzzles) JPwLEG-3 150-piece puzzles (SJP dataset) MovingMNIST CLEVRER UCF101 QST
Metrics: puzzle-level accuracy piece-level accuracy normalized Kendall distance
Baselines: SAJ Deepzzle SJP-ED Siamese Puzzlet STP VCOP J-VAD
Builds on: denoising diffusion probabilistic models masked autoencoders Space-Time Puzzle
Closest papers by meaning
- ERL-MPP: Evolutionary Reinforcement Learning with Multi-head Puzzle Perception for Solving Large-scale Jigsaw Puzzles of Eroded Gaps (AAAI 2025 · similarity 0.83)
- Puzzles: Unbounded Video-Depth Augmentation for Scalable End-to-End 3D Reconstruction (NeurIPS 2025 · similarity 0.83)
- VDT: General-purpose Video Diffusion Transformers via Mask Modeling (ICLR 2024 · similarity 0.83)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.82)
- Visual Jigsaw Post-Training Improves MLLMs (ICLR 2026 · similarity 0.82)
- NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.