Research map / Large language models / Language, safety and interpretability
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, Jifeng Dai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InternImage-H instead of ResNet-50 as the visual backbone | VisionLLM · COCO2017 detection | AP: 44.6 -> 60.2 | Improved |
| Replace query-based image tokenization with average pooling (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.8 -> 23.1 | Hurt |
| Add a BERT text encoder (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.7 -> 44.8 | No clear effect |
| Add a BERT text encoder (ablation) | VisionLLM-R50 · RefCOCO visual grounding | P@0.5: 48.1 -> 84.1 | Improved |
| Freeze the BERT text encoder during training (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.8 -> 1.3 | Hurt |
| Freeze the BERT text encoder during training (ablation) | VisionLLM-R50 · RefCOCO visual grounding | P@0.5: 84.1 -> 34.3 | Hurt |
| Increase localization vocabulary from 257 to 1025 bins (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 34.9 -> 44.8; +9.9 AP | Improved |
| Increase localization vocabulary from 1025 to 2049 bins (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.8 -> 44.8 | No clear effect |
| Train vision-centric tasks jointly instead of separately (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.8 -> 44.6 | Hurt |
| Add random descriptions, categories, and output formats during training (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 45.2 -> 44.6 | Hurt |
| Use LoRA when training with randomness (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 1.2 -> 44.8 | Improved |
| Increase image tokens from 50 to 100 (ablation) | VisionLLM-R50 · COCO2017 detection | AP: 44.5 -> 44.8 | Improved |
Closest papers by meaning
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.92)
- Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model (CVPR 2024 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models (NeurIPS 2024 · similarity 0.89)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
