Research map / Vision-language and multimodal / Visual question answering
Foundation Model is Efficient Multimodal Multitask Model Selector
Fanqing Meng, Wenqi Shao, zhanglin peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, Ping Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of F-Labels K from 1 to 3 (ablation) | EMMS · Flickr8k captioning | weighted Kendall tau: 0.490 -> 0.660 | Improved |
| increase K from 3 to 4 (ablation) | EMMS · FlickrStyle10K-Romantic captioning | weighted Kendall tau: 0.678 -> 0.644 | Hurt |
| increase K from 3 to 4 (ablation) | EMMS · Flickr8k captioning | weighted Kendall tau: 0.660 -> 0.660 | No clear effect |
| reduce alternating-minimization iterations r from 3 to 1 (ablation) | EMMS · DAQUAR VQA | weighted Kendall tau: 0.743 -> 0.712 | No clear effect |
| reduce iterations r from 3 to 1 (ablation) | EMMS · CLEVR VQA | wall-clock time (s): 745.11 -> 274.56 | Improved |
| use CLIP-L instead of CLIP-B for F-Label (ablation) | EMMS · FlickrStyle10K-Humor captioning | weighted Kendall tau: 0.480 -> 0.802 | Improved |
| replace one-hot label with F-Label (ablation) | EMMS · CNN image classification avg | weighted Kendall tau: 0.618 -> 0.664 | Improved |
| replace one-hot label with F-Label (ablation) | EMMS · ViT image classification avg | weighted Kendall tau: 0.611 -> 0.639 | Improved |
| replace one-hot label with F-Label (ablation) | EMMS · CNN image classification avg | wall-clock time (s): 42.99 -> 143.2 | Hurt |
| combine three foundation models instead of single GPT-2 (ablation) | EMMS · image classification avg | weighted Kendall tau: 0.610 -> 0.639 | Improved |
| combine three foundation models instead of single GPT-2 (ablation) | EMMS · image captioning avg | weighted Kendall tau: 0.545 -> 0.670 | Improved |
| combine three foundation models instead of CLIP alone (ablation) | EMMS · image captioning avg | weighted Kendall tau: 0.628 -> 0.670 | Improved |
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.87)
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.86)
- Ranked from Within: Ranking Large Multimodal Models Without Labels (ICML 2025 · similarity 0.86)
- Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE (NeurIPS 2024 · similarity 0.86)
- Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.86)
- Generative Multimodal Models are In-Context Learners (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
