Research map / Vision-language and multimodal / Human and embodied modeling
Seeing the Abstract: Translating the Abstract Language for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fashion language is abstract-heavy and that CLIP-style VLMs under-represent it, then proposes ACT, a training-free method that LLM-rewrites abstract queries and shifts their text embeddings toward concrete ones using a PCA direction learned from paired abstract/caption data.
Problem
Vision-language models are pre-trained mostly on concrete, visually grounded web captions, so they under-represent abstract-oriented language (feelings, style, mood) that dominates fashion descriptions, hurting text-to-image retrieval.
Approach
Build paired abstract and concrete (captioner-generated) descriptions of the same images, PCA the text-embedding difference to get a shift subspace, then at inference rephrase abstract queries with an LLM and add the projected shift to the text embedding, with no training.
Key results
- ACT outperforms zero-shot VLMs by up to +12.6% H@1 on DeepFashion text-to-image retrieval.
- Training-free ACT-df reaches H@1 0.437 vs 0.417 for fine-tuned SigLIP-ft-df (about +2.0%).
- Across backbones, ACT gives an average +4.9% H@1 gain; ablation shows language rewriting adds about 10% H@1 and representation shift about 3.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ACT instead of fine-tuning SigLIP on DeepFashion | SigLIP (ViT-B-16) · DeepFashion, same-dataset | H@1: .417 -> .437 | Helped |
| Apply ACT instead of fine-tuning SigLIP on FACAD | SigLIP (ViT-B-16) · FACAD → DeepFashion, cross-dataset | H@1: .352 -> .428 | Helped |
| Add language rewriting while omitting representation shift (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .311 -> .411 | Helped |
| Add representation shift while omitting language rewriting (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .311 -> .347 | Helped |
| Add representation shift to Llama 3.1-8B language rewriting (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .411 -> .437 | Helped |
| Replace Qwen2-VL captions with CogVLM2 captions (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .437 -> .433 | Hurt |
| Replace caption text with image embeddings for representation shift analysis (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .437 -> .406 | Hurt |
| Replace Llama 3.1-8B with Llama 3.2-3B for language rewriting (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .437 -> .365 | Hurt |
| Replace Llama 3.1-8B with Phi3-mini-4K for language rewriting (ablation) | ACT on SigLIP (ViT-B-16) · DeepFashion | H@1: .437 -> .424 | Hurt |
Surprising findings
- Current general-purpose and fashion-specific VLMs retrieve better with concrete captions than with the original abstract fashion descriptions.
- Shifting with visual embeddings instead of text-derived concrete information degrades performance, possibly due to the modality gap.
Limitations
- Analysis is limited to fashion and to abstract expressions identified via adjectives.
- Relies on an LLM rewriter and a captioner, so quality depends on those models and may still contain abstract words.
- Gains on Recall@K are smaller than on Hit-Rate@K in some settings.
Experimental setup
Datasets: DeepFashion FACAD LAION 400M
Benchmarks: DeepFashion FACAD
Metrics: Recall@K Hit-Rate@K
Baselines: SigLIP CLIP O-CLIP EVA-CLIP F-CLIP OF-CLIP SigLIP-ft-df F-CLIP-ft-df OF-CLIP-ft-df
Builds on: CLIP SigLIP F-CLIP OF-CLIP
Closest papers by meaning
- Easy Regional Contrastive Learning of Expressive Fashion Representations (NeurIPS 2024 · similarity 0.89)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.87)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.