AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data

Maya Varma, Jean-Benoit Delbrouck, Sarah Hooper, Akshay Chaudhari, Curtis Langlotz

ICCV 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 17 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase training data pairwise complexity from 5.0 to 29.4 in a one-to-one VLM (ablation)CLIP-RN50 one-to-one VLM · DocMNIST text->region retrievalR-Precision: drops 36.9% (32.4 R-Precision points)Hurt
increase training data pairwise complexity from 5.0 to 29.4 in a one-to-one VLM (ablation)CLIP-RN50 one-to-one VLM · DocMNIST region->text retrievalR-Precision: drops 20.5% (20.0 R-Precision points)Hurt
train a one-to-one VLM on region-attribute pairs from a learned mapping model instead of image-text pairs onlyCLIP-RN50 · DocMNIST text->region retrievalR-Precision: 55.2 -> 69.4Improved
learn region-attribute mappings with a trained mapping model instead of zero-shot CLIP (ablation)ViLLA pipeline · DocMNIST text->region retrievalR-Precision: 35.2 -> 69.4Improved
learn region-attribute mappings with a trained mapping model instead of zero-shot CLIP (ablation)ViLLA pipeline · DeepFashion text->region retrievalR-Precision: 35.3 -> 39.5Improved
align each region to the whole text description (CLIP-FT-Reg) instead of image-level fine-tuning (ablation)CLIP fine-tuning · DocMNIST text->region retrievalR-Precision: 55.2 -> 29.0Hurt
apply ViLLA on a lower-complexity dataset (DeepFashion vs DocMNIST)ViLLA · retrievalR-Precision gain over next best baseline: 14.2 vs 4.2 R-Precision pointsMixed
train ViLLA on COCO instead of using RegionCLIPRegionCLIP baseline · COCO zero-shot object detectionAP50 (All): 62.8 -> 66.4Improved
train ViLLA instead of RegionCLIP on LVIS rare/common classesRegionCLIP baseline · LVIS zero-shot object detectionAPc: 43.5 -> 42.4Mixed
use ViLLA one-to-one VLM instead of GLoRIA local+global lossesCheXpert 5x200 retrieval · CheXpert 5x200 retrievalaccuracy: 52.1 -> 55.9Improved
add GLoRIA local loss to global loss (ablation)GLoRIA · CheXpert 5x200 retrievalaccuracy: 52.1 -> 48.8Hurt
learn mapping model instead of zero-shot VLM mapping (ablation)region-attribute mapping · DocMNIST region-attribute mappingF1: 42.6 -> 68.4Improved

Later papers that compared with this method

1 later accepted paper reports VILLA as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy