Research map / Vision-language and multimodal / Multimodal LLMs
How Can Objects Help Video-Language Understanding?
Zitian Tang, Shijie Wang, Junho Cho, Jaewook Yoo, Chen Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent boxes as quantized plain text instead of a learned embedding projector (ablation) | ObjectMLLM · CLEVRER-MC | accuracy: 64.9 -> 77.6 | Improved |
| represent boxes as plain text instead of embedding projector, with 10k training samples (ablation) | ObjectMLLM · CLEVRER-MC (10k samples) | accuracy: 44.5 -> 63.8 | Improved |
| zero-initialize the box embedding projector instead of random init (ablation) | Embedding projector · CLEVRER-MC | accuracy: 42.8 -> 64.9 | Improved |
| use MLP with more layers for box embedding projector (ablation) | Embedding projector · CLEVRER-MC | accuracy: 64.9 -> 65.0 | No clear effect |
| use 9 tokens per box in embedding projector instead of 1 (ablation) | Embedding projector · CLEVRER-MC | accuracy: 64.9 -> 63.4 | Hurt |
| add object bounding boxes to captions (ablation) | ObjectMLLM (LLaMA3) · STAR | accuracy: 60.1 -> 64.4 | Improved |
| add object bounding boxes to captions (ablation) | ObjectMLLM (LLaMA3) · CLEVRER-MC | accuracy: 47.8 -> 75.5 | Improved |
| add object bounding boxes to captions (ablation) | ObjectMLLM (LLaMA3) · NExT-QA and IntentQA | accuracy: 76.6 -> 76.6 (NExT-QA); 75.7 -> 75.6 (IntentQA) | No clear effect |
| add CLIP visual embeddings on top of caption and box (ablation) | ObjectMLLM (LLaMA3) · CLEVRER-MC | accuracy: 75.5 -> 75.4 | No clear effect |
| add boxes to pre-trained VideoLLaMA2 and LoRA fine-tune | VideoLLaMA2-7B · CLEVRER-MC | accuracy: 67.9 -> 77.6 | Improved |
| add boxes to VideoLLaMA2 without fine-tuning (zero-shot) | VideoLLaMA2-7B · CLEVRER-MC | accuracy: 45.6 -> 34.4 | Hurt |
| train modality-by-modality instead of jointly on all modalities (ablation) | ObjectMLLM (LLaMA3) · CLEVRER-MC | accuracy: 29.5 -> 75.4 | Improved |
Closest papers by meaning
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
