Research map / Vision-language and multimodal / CLIP and zero-shot
ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data
Maya Varma, Jean-Benoit Delbrouck, Sarah Hooper, Akshay Chaudhari, Curtis Langlotz
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase training data pairwise complexity from 5.0 to 29.4 in a one-to-one VLM (ablation) | CLIP-RN50 one-to-one VLM · DocMNIST text->region retrieval | R-Precision: drops 36.9% (32.4 R-Precision points) | Hurt |
| increase training data pairwise complexity from 5.0 to 29.4 in a one-to-one VLM (ablation) | CLIP-RN50 one-to-one VLM · DocMNIST region->text retrieval | R-Precision: drops 20.5% (20.0 R-Precision points) | Hurt |
| train a one-to-one VLM on region-attribute pairs from a learned mapping model instead of image-text pairs only | CLIP-RN50 · DocMNIST text->region retrieval | R-Precision: 55.2 -> 69.4 | Improved |
| learn region-attribute mappings with a trained mapping model instead of zero-shot CLIP (ablation) | ViLLA pipeline · DocMNIST text->region retrieval | R-Precision: 35.2 -> 69.4 | Improved |
| learn region-attribute mappings with a trained mapping model instead of zero-shot CLIP (ablation) | ViLLA pipeline · DeepFashion text->region retrieval | R-Precision: 35.3 -> 39.5 | Improved |
| align each region to the whole text description (CLIP-FT-Reg) instead of image-level fine-tuning (ablation) | CLIP fine-tuning · DocMNIST text->region retrieval | R-Precision: 55.2 -> 29.0 | Hurt |
| apply ViLLA on a lower-complexity dataset (DeepFashion vs DocMNIST) | ViLLA · retrieval | R-Precision gain over next best baseline: 14.2 vs 4.2 R-Precision points | Mixed |
| train ViLLA on COCO instead of using RegionCLIP | RegionCLIP baseline · COCO zero-shot object detection | AP50 (All): 62.8 -> 66.4 | Improved |
| train ViLLA instead of RegionCLIP on LVIS rare/common classes | RegionCLIP baseline · LVIS zero-shot object detection | APc: 43.5 -> 42.4 | Mixed |
| use ViLLA one-to-one VLM instead of GLoRIA local+global losses | CheXpert 5x200 retrieval · CheXpert 5x200 retrieval | accuracy: 52.1 -> 55.9 | Improved |
| add GLoRIA local loss to global loss (ablation) | GLoRIA · CheXpert 5x200 retrieval | accuracy: 52.1 -> 48.8 | Hurt |
| learn mapping model instead of zero-shot VLM mapping (ablation) | region-attribute mapping · DocMNIST region-attribute mapping | F1: 42.6 -> 68.4 | Improved |
Later papers that compared with this method
1 later accepted paper reports VILLA as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Think Before You Drive: World Model-Inspired Multimodal Grounding (CVPR 2026 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples (AAAI 2025 · similarity 0.89)
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)
- RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
