Research map / Vision-language and multimodal / Human and embodied modeling
SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyncMask, which uses momentum-model cross-attention to pick co-occurring image patches and text tokens for masked modeling, plus semi-hard negative grouped batch sampling, for fashion vision-language pretraining.
Problem
Fashion image-text datasets pair multiple images of one item with one text, so random masking in MLM and MIM masks non-co-occurring elements and false negatives arise in ITC/ITM batches, hurting fine-grained cross-modal alignment.
Approach
Replace random masks in MLM and MIM with masks selected from the momentum (teacher) model's vision-language cross-attention so that masked elements co-occur in image and text, and group batches with semi-hard negatives while excluding same-item positives to reduce false negatives.
Key results
- On FashionGen full-set retrieval, SyncMask reaches 73.00 mean R@1 for image-to-text/text-to-image on the subset (75.00 I2T, 71.00 T2I), above FashionSAP's 71.63.
- On FashionIQ text-guided image retrieval, SyncMask reports 38.13 mean R@10/R@50 average, above FashionSAP's 36.26 (mean R@10 across dresses, toptees, shirts).
- Ablation shows combining textual and visual attentional masking gives +2.14 mean gain over all-random masking, and semi-hard grouping gives +2.46 over random grouping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random text and image masks with synchronized attentional masks (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 70.31 -> 72.45 (+2.14) | Helped |
| Use attentional text masking with random visual masking (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 70.31 -> 70.71 (+0.40) | Helped |
| Use attentional visual masking with random text masking (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 70.31 -> 70.97 (+0.66) | Helped |
| Group hardest rather than random negatives in batches (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 69.99 -> 71.89 (+1.90) | Helped |
| Exclude false negatives during hardest-negative grouping (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 69.99 -> 71.97 (+1.98) | Helped |
| Use semi-hard instead of hardest negatives while excluding false negatives (ablation) | SyncMask · FashionGen downstream tasks | I2T R@1: 69.99 -> 72.45 (+2.46) | Helped |
| Apply SyncMask rather than random masking | SyncMask · FashionGen sub-set retrieval | I2T R@1: 73.14 -> 75.00 | Helped |
| Apply SyncMask and semi-hard negative sampling | SyncMask · FashionIQ mean | R@10: 36.26 -> 38.13 | Helped |
| Apply SyncMask and semi-hard negative sampling | SyncMask · FashionGen subcategory recognition | accuracy: 94.33 -> 94.21 | Hurt |
Surprising findings
- Hardest-negative grouping without false-negative exclusion gave a smaller gain (+1.90) than semi-hard grouping, suggesting that the hardest negatives are often false negatives in fashion data.
Limitations
- Gains on several tasks are small (e.g., about 0.4-2 points), and results are from single runs with no variance reported.
- Relies on a momentum model's cross-attention, so mask quality depends on the teacher model's current state.
- Semi-hard sampling depends on a predefined hyperparameter s, and false negatives that cannot be excluded by item index remain.
Experimental setup
Datasets: FashionGen FashionIQ
Benchmarks: FashionGen FashionIQ
Metrics: R@1 R@10 R@50 Macro-F1 Accuracy
Baselines: VSE++ VL-BERT ViLBERT Image-BERT OSCAR FashionBERT Kaleido-BERT CommerceMM EI-CLIP ALBEF FashionViL FashionSAP
Builds on: ALBEF GRIT-VLP FashionSAP MoCo Kaleido-BERT
Closest papers by meaning
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.87)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.87)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.87)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.87)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.87)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.