Research map / Vision-language and multimodal / Human and embodied modeling
Conditional Representation Learning for Customized Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conditional Representation Learning (CRL), which uses an LLM to generate descriptive texts for a user-specified criterion and projects frozen VLM image embeddings onto that text basis to obtain criterion-tailored representations, evaluated on few-shot, clustering, retrieval, and fashion retrieval tasks.
Problem
Universal image representations capture dominant semantics (e.g., shape or category) and may not suit customized downstream tasks that need other criteria (e.g., color, scene, texture), while supervised fine-tuning for each criterion is costly in annotation and compute.
Approach
Treat a set of LLM-generated descriptive words for a criterion as a semantic basis; project VLM image embeddings onto the text embeddings of that basis (R = I T^T) to obtain a training-free conditional representation.
Key results
- CRL on CLIP improves mean few-shot accuracy from 52.01 to 64.96 on Clevr4-10k, and BLIP2+CRL reaches 51.75 vs 43.73 on Cards.
- On customized clustering, BLIP2+CRL reaches 43.98 mean ACC/NMI/ARI-style score on Cards and Clevr4-10k vs 20.98 for BLIP2, and CLIP+CRL gains about 75% ACC on color.
- On GeneCIS similarity retrieval, CLIP+CRL improves mean recall from 17.0 to 26.8 (training-free), and CRL with fine-tuned CLIP reaches 32.6 vs Combiner's 28.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CRL-conditioned representations to ALIGN | ALIGN · Clevr4-10k customized few-shot learning | mean accuracy: 53.00 -> 67.76 | Helped |
| Add CRL-conditioned representations to BLIP2 | BLIP2 · Cards customized few-shot learning | mean accuracy: 43.73 -> 51.75 | Helped |
| Add CRL-conditioned representations to MetaCLIP | MetaCLIP · Clevr4-10k customized clustering | mean: 28.46 -> 62.23 | Helped |
| Add CRL-conditioned representations to BLIP2 | BLIP2 · Cards customized clustering | mean: 20.98 -> 43.98 | Helped |
| Apply CRL using fine-tuned CLIP weights | CLIP on GeneCIS · GeneCIS customized similarity retrieval | mean recall: 28.9 -> 32.6 | Helped |
| Use training-free CRL representations instead of CLIP image embeddings | CLIP on DeepFashion · DeepFashion customized fashion retrieval | mean MAP: 6.08 -> 7.93 | Helped |
| Train the CRL representation and backbone | CLIP+CRL on DeepFashion · DeepFashion customized fashion retrieval | mean MAP: 10.22 -> 11.33 | Helped |
| Increase the number of generated basis texts from 10 to 200 (ablation) | CRL on Clevr4-10k customized few-shot learning · Clevr4-10k customized few-shot learning | Mean accuracy: 52.09 -> 65.64 | Helped |
| Vary the number of generated basis texts from 10 to 200 (ablation) | CRL on Clevr4-10k customized clustering · Clevr4-10k customized clustering | Mean: 43.25 -> 58.47 | Helped |
| Increase the number of generated basis texts from 10 to 200 (ablation) | CRL on Cards customized clustering · Cards customized clustering | Mean: 19.64 -> 23.90 | Helped |
| Increase the number of generated basis texts from 10 to 200 (ablation) | CRL on GeneCIS · GeneCIS customized similarity retrieval | Mean recall: 28.6 -> 32.4 | Helped |
Surprising findings
- Traditional clustering methods CC and SCAN outperform CRL on the universal 'shape' criterion, which the authors attribute to specialized clustering designs.
Limitations
- May not outperform specialized clustering methods (CC, SCAN) on the universal 'shape' criterion.
- The LLM-generated basis is only a rough approximation and filtering strategies were not effective across all criteria.
- No error bars or statistical significance tests reported due to compute constraints.
Experimental setup
Datasets: Clevr4-10k Cards GeneCIS CoCo DeepFashion
Benchmarks: Clevr4-10k Cards GeneCIS DeepFashion
Metrics: accuracy NMI ARI recall@k MAP
Baselines: CLIP ALIGN MetaCLIP BLIP2 CC SCAN Multi-Map Pic2Word SEARLE LinCIR CIG Combiner
Builds on: CLIP Multi-Map Pic2Word Taskonomy
Closest papers by meaning
- Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements (CVPR 2024 · similarity 0.84)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.84)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.84)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.84)
- Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.83)
- Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.