Research map / Vision-language and multimodal / CLIP and zero-shot
AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs
Yuan Zhang, Chun-Kai Fan, Sicheng Yu, Junwen Pan, Tao Huang, Ming Lu, Kuan Cheng, Qi She, Shanghang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retrieve a visual prompt with AutoV | LLaVA-OneVision 7B · VizWiz | score: 58.2 -> 68.4 | Improved |
| retrieve a visual prompt with AutoV | Qwen2.5-VL 7B · MMMU | score: 50.1 -> 53.9 | Improved |
| fixed FGVP prompt | LLaVA-1.5 7B · TextVQA | score: 58.2 -> 46.3 | Hurt |
| pairwise ranking instead of regression (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 36.9 -> 38.7 | Improved |
| pairwise instead of list-wise ranking (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 38.0 -> 38.7 | Improved |
| pairwise ranking instead of MoE GateNet (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 37.6 -> 38.7 | Improved |
| random retrieval vs baseline (ablation) | AutoV pool · Avg of MMMU/VizWiz/MMVet | score: +0.7 | Improved |
| use 0th LLM layer for modality interaction instead of 24th (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 38.1 -> 38.7 | Improved |
| add pre-filtering of farthest prompt (6 candidates) (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 38.3 -> 38.7 | Improved |
| add pre-filtering (8 candidates) (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 38.4 -> 38.9 | Improved |
| enlarge candidate pool from 6 to 7 prompts (ablation) | AutoV · MMMU (LLaVA-1.5 7B) | score: 38.7 -> 38.7 | No clear effect |
| enlarge candidate pool from 1 to 8 prompts (ablation) | AutoV · Avg of MMMU/VizWiz/MMVet | score: 39.8 -> 41.6 | Improved |
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.90)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
