Research map / Vision-language and multimodal / CLIP and zero-shot
Easy Regional Contrastive Learning of Expressive Fashion Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP to fashion retrieval by adding selection tokens and fusion blocks to the image encoder, trained with contrastive and region contrastive losses, and releases a new AmazonFashion dataset.
Problem
Fine-tuned CLIP on fashion data ignores small but critical details (logos, composition), hurting cross-modal retrieval and captioning, while existing fashion VLMs use complex BERT-based architectures with multiple objectives.
Approach
Insert a few tag-associated selection tokens and hard-attention fusion blocks (Gumbel-softmax straight-through) into CLIP's ViT image encoder, supervised by region contrastive losses against tag entities, keeping only the contrastive objective and adding ~1.9% parameters.
Key results
- On FashionGen full evaluation, E2 reaches SumR 497.9 versus 468.2 for CLIP-FT (image-to-text R@1 62.8 vs 55.2).
- On AmazonFashion full evaluation, E2 achieves SumR 112.2 (sample-100) versus 99.5 for CLIP-FT, and 7.5 vs 6.1 image-to-text R@1 in full evaluation.
- Plugging E2 into DeCap raises zero-shot captioning B@4 from 16.88 to 21.35 on FashionGen; CIReVL with E2 improves average R@10 on Fashion-IQ from 28.29 to 31.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add E2 regional selection, fusion, and region contrastive learning to CLIP | CLIP ViT-B-32 · FashionGen full evaluation, image-to-text retrieval | R@1: 55.5 -> 62.8 | Helped |
| Add E2 regional selection, fusion, and region contrastive learning to CLIP | CLIP ViT-B-32 · AmazonFashion full evaluation, image-to-text retrieval | R@1: 6.2 -> 7.5 | Helped |
| Replace DeCap's CLIP backbone with E2 | DeCap · FashionGen zero-shot image captioning | B@4: 16.88 -> 21.35 | Helped |
| Replace CIReVL's CLIP encoders with E2 | CIReVL · Fashion-IQ Average | R@10: 28.29 -> 31.73 | Helped |
| Add fusion blocks (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 58.8 -> 62.8 | Helped |
| Add selection tokens (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 55.2 -> 62.8 | Helped |
| Add region contrastive loss (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 56.3 -> 62.8 | Helped |
| Use a FILIP backbone instead of a CLIP backbone (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 62.8 -> 62.1 | Hurt |
| Add Composition selection tokens (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 59.4 -> 62.8 | Helped |
| Add Sub-category selection tokens (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 61.2 -> 62.8 | Helped |
| Add Brand selection tokens (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 58.3 -> 62.8 | Helped |
| Add Season selection tokens (ablation) | E2 · FashionGen full evaluation, image-to-text retrieval | R@1: 60.0 -> 62.8 | Helped |
Surprising findings
- Ablating the selection tokens for Season and Sub-category yields smaller gains than Brand and Composition, which the authors attribute to these being easier to distinguish visually.
- Replacing the Fusion Blocks alone (keeping selection tokens and region loss) still leaves most of the gain over CLIP-FT, suggesting implicit interaction via ViT layers contributes substantially.
Limitations
- Performance depends on the quality of available tag entities in fine-tuning data.
- Inherits limitations of the pre-trained CLIP foundation model's general-domain pre-training.
- Evaluated mainly on FashionGen and a self-collected AmazonFashion dataset, with limited scope beyond cross-modal retrieval.
Experimental setup
Datasets: FashionGen AmazonFashion FashionIQ
Benchmarks: FashionGen AmazonFashion Fashion-IQ
Metrics: Recall@1 Recall@5 Recall@10 SumR BLEU@4 CIDEr METEOR ROUGE
Baselines: CLIP-FT FILIP-FT FashionSAP FashionViL FashionCLIP EI-CLIP FAME-ViL CIReVL DeCap CapDec
Builds on: CLIP FILIP DeCap CapDec CIReVL
Closest papers by meaning
- FashionERN: Enhance-and-Refine Network for Composed Fashion Image Retrieval (AAAI 2024 · similarity 0.90)
- Seeing the Abstract: Translating the Abstract Language for Vision Language Models (CVPR 2025 · similarity 0.89)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.87)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.87)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.86)
- Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.