AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, Alex Kot

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 44 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add Style Prompt Enhancement (predict style before answering)LLaVA-1.5-13B · BenchLMM artistic stylesaverage accuracy: 59.3 -> 65.1Improved
add Style Prompt EnhancementInstructBLIP-13B · BenchLMM artistic stylesaverage accuracy: 58.9 -> 62.5Improved
add Style Prompt EnhancementLLaVA-1.5-13B · BenchLMM sensor stylesaverage accuracy: 47.3 -> 51.8Improved
add Style Prompt EnhancementInstructBLIP-13B · BenchLMM sensor stylesaverage accuracy: 37.3 -> 38.7Improved
add Style Prompt EnhancementLLaVA-1.5-13B · BenchLMM application stylesaverage accuracy: 45.2 -> 46.6Improved
add Style Prompt EnhancementInstructBLIP-13B · BenchLMM application stylesaverage accuracy: 38.9 -> 40.3Improved
add Style Prompt EnhancementLLaVA-1.5-13B · BenchLMM style-transfer images (DALL-E3)average accuracy: 57.1 -> 60.8Improved
add Style Prompt EnhancementInstructBLIP-13B · BenchLMM style-transfer images (DALL-E3)average accuracy: 64.0 -> 67.2Improved
evaluate on non-common styles instead of common photo styleGPT-4V · BenchLMMaccuracy: 81.5 -> 55.0 (Infrared)Hurt
use 13B instead of 7B language modelInstructBLIP · BenchLMM artistic stylesaccuracy: Common 73.9 -> 72.7; artistic avg 55.3 -> 58.9Mixed
SPE applied to application-style data vs artistic/sensor dataSPE · BenchLMM application stylesaverage accuracy gain: +1.4 vs +5.8 (artistic)Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy