Research map / Vision-language and multimodal / CLIP and zero-shot
RILS: Masked Visual Reconstruction in Language Semantic Space
Shusheng Yang, Yixiao Ge, Kun Yi, Dian Li, Ying Shan, Xiaohu Qie, Xinggang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| perform masked reconstruction using language-semantic text prototypes instead of raw pixels or learnable vision-space prototypes (ablation) | RILS · ImageNet-1K, L-10M | zero-shot accuracy: 34.2 -> 37.5; 34.8 -> 37.5 | Improved |
| combine MIM and image-text contrastive learning in an end-to-end language-space reconstruction framework rather than two-stage training | RILS · ImageNet-1K, L-10M | linear probing accuracy: 64.8 -> 68.5 | Improved |
| reduce the mask ratio to 60% (ablation) | RILS · ImageNet-1K, L-10M | linear probing accuracy: 68.5 -> 69.3 | Improved |
| reduce the mask ratio to 60% (ablation) | RILS · ImageNet-1K, L-10M | zero-shot accuracy: 37.5 -> 35.9 | Hurt |
| increase the mask ratio to 90% (ablation) | RILS · ImageNet-1K, L-10M | linear probing accuracy: 68.5 -> 67.6 | Hurt |
| increase the mask ratio to 90% (ablation) | RILS · ImageNet-1K, L-10M | fine-tuning accuracy: 82.7 -> 82.3 | Hurt |
| increase decoder depth from 1 to 4 blocks (ablation) | RILS · ImageNet-1K, L-10M | linear probing accuracy: 68.5 -> 69.6 | Improved |
| increase decoder depth from 1 to 4 blocks (ablation) | RILS · ImageNet-1K, L-10M | fine-tuning accuracy: 82.7 -> 82.5 | Hurt |
| increase decoder depth from 1 to 4 blocks (ablation) | RILS · ImageNet-1K, L-10M | zero-shot accuracy: 37.5 -> 37.3 | No clear effect |
| increase contrastive-to-reconstruction loss ratio from 1:1 to 2:1 (ablation) | RILS · ImageNet-1K, L-10M | zero-shot accuracy: 36.4 -> 37.5 | Improved |
| increase contrastive-to-reconstruction loss ratio from 1:2 to 2:1 (ablation) | RILS · ImageNet-1K, L-10M | zero-shot accuracy: 35.9 -> 37.5 | Improved |
| scale pre-training data from LAION-10M to LAION-100M | RILS · ImageNet-1K | zero-shot accuracy: 37.5 -> 50.6 | Improved |
Closest papers by meaning
- Learning Visual Representations with Caption Annotations (ECCV 2020 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · similarity 0.88)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.88)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.87)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
