AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

BRAVE: Broadening the visual encoding of vision-language models

Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, Federico Tombari

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 61 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
combine five frozen vision encoders with MEQ-Former instead of one encoderBRAVE · MMVP testaverage pair accuracy: 27.3 (best single, EVA-CLIP-g) -> 42.0Improved
replace ensemble of Q-Formers with a single MEQ-Former (ablation)BRAVE · COCO captioning (224 res)CIDEr: 140.9 -> 145.2Improved
fine-tune the LM for VQA instead of frozen LM (ablation)BRAVE · VQAv2 Karpathy valaccuracy: 78.6 -> 81.8Improved
LoRA rank 128 instead of full LM fine-tuning (ablation)BRAVE · VQAv2 Karpathy valaccuracy: 81.8 -> 81.0Hurt
add synthetic VQA data (ablation)BRAVE · OKVQA valaccuracy: 64.0 -> 65.7Improved
add encoder dropout during pre-training (ablation)BRAVE · COCO captioning Karpathy valCIDEr: 145.3 -> 147.0Improved
add encoder dropout during pre-training (ablation)BRAVE · OKVQA valaccuracy: 66.0 -> 65.7Hurt
add high-resolution fine-tuning stage (ablation)BRAVE · VQAv2 Karpathy valaccuracy: 79.6 -> 81.8Improved
use FlanT5-XL instead of FlanT5-L (ablation)BRAVE · COCO captioning Karpathy valCIDEr: 142.5 -> 147.0Improved
pre-train on WebLI instead of CC3M (ablation)BRAVE · COCO captioning (224 res)CIDEr: 138.3 -> 145.2Improved
scale single vision encoder from ViT-G (1.8B) to ViT-e (3.8B)Q-Former VLM · VQAv2 Karpathy valaccuracy: 74.2 -> 75.6Improved
use OpenCLIP-G (larger, more data) instead of CLIP-L/14Q-Former VLM · COCO captioning Karpathy valCIDEr: 133.0 -> 128.3Hurt

Later papers that compared with this method

2 later accepted papers report BRAVE as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy