Research map / Vision-language and multimodal / Human and embodied modeling
A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models
Weixin Ye, Wei Wang, Yahui Liu, Yue Song, Bin Ren, Wei Bi, Rita Cucchiara, Nicu Sebe
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MJP (shuffle + unk position embedding + DAL) to DeiT-S | DeiT-S · ImageNet-1K val | Top-1 accuracy: 79.8 -> 80.5 | Improved |
| add MJP to Swin-T | Swin-T · ImageNet-1K val | Top-1 accuracy: 81.3 -> 81.3 | No clear effect |
| add MJP to ResNet-50 (patch shuffle only) | ResNet-50 · ImageNet-1K val | Top-1 accuracy: 79.3 -> 79.4 | No clear effect |
| add MJP to DeiT-B | DeiT-B · ImageNet-1K val | Top-1 accuracy: 81.8 -> 82.0 | Improved |
| use masking ratio 0.03 instead of 0 when training DeiT-S (ablation) | DeiT-S + MJP · ImageNet-1K val | Top-1 accuracy: 80.0 -> 80.5 | Improved |
| add MJP to Swin-Tiny backbone for UperNet | Swin-Tiny · ADE20K segmentation | mIoU: 43.87 -> 44.03 | Improved |
| add MJP instead of naive shuffle to BERT-base | BERT-base · Yelp sentiment | Top-1 accuracy: 70.34 -> 70.68 | Improved |
| add MJP instead of naive shuffle to BERT-base | BERT-base · Amazon sentiment | Top-1 accuracy: 72.03 -> 72.17 | Improved |
| add MJP to Transformer4 | Transformer4 · Yelp sentiment | Top-1 accuracy: 57.89 -> 58.05 | Improved |
| use n-gram window size 32 rather than 128 for MJP (ablation) | BERT-base + MJP · Yelp sentiment | Top-1 accuracy: 70.25 -> 70.68 | Mixed |
| add DAL/DRL auxiliary position loss to MJP for text (ablation) | BERT-base + MJP · Yelp sentiment | Top-1 accuracy | No clear effect |
| raise masking ratio to 0.7-0.8 for BERT-base on Yelp (ablation) | BERT-base + MJP · Yelp sentiment | Top-1 accuracy: 70.68 -> 70.07 | Hurt |
Closest papers by meaning
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.85)
- DAGER: Exact Gradient Inversion for Large Language Models (NeurIPS 2024 · similarity 0.85)
- Stochastic positional embeddings improve masked image modeling (ICML 2024 · similarity 0.85)
- Maximizing the Position Embedding for Vision Transformers with Global Average Pooling (AAAI 2025 · similarity 0.84)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.84)
- Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
