AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, Jifeng Dai

NeurIPS 2023 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 132 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use InternImage-H instead of ResNet-50 as the visual backboneVisionLLM · COCO2017 detectionAP: 44.6 -> 60.2Improved
Replace query-based image tokenization with average pooling (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.8 -> 23.1Hurt
Add a BERT text encoder (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.7 -> 44.8No clear effect
Add a BERT text encoder (ablation)VisionLLM-R50 · RefCOCO visual groundingP@0.5: 48.1 -> 84.1Improved
Freeze the BERT text encoder during training (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.8 -> 1.3Hurt
Freeze the BERT text encoder during training (ablation)VisionLLM-R50 · RefCOCO visual groundingP@0.5: 84.1 -> 34.3Hurt
Increase localization vocabulary from 257 to 1025 bins (ablation)VisionLLM-R50 · COCO2017 detectionAP: 34.9 -> 44.8; +9.9 APImproved
Increase localization vocabulary from 1025 to 2049 bins (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.8 -> 44.8No clear effect
Train vision-centric tasks jointly instead of separately (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.8 -> 44.6Hurt
Add random descriptions, categories, and output formats during training (ablation)VisionLLM-R50 · COCO2017 detectionAP: 45.2 -> 44.6Hurt
Use LoRA when training with randomness (ablation)VisionLLM-R50 · COCO2017 detectionAP: 1.2 -> 44.8Improved
Increase image tokens from 50 to 100 (ablation)VisionLLM-R50 · COCO2017 detectionAP: 44.5 -> 44.8Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy