Research map / Vision-language and multimodal / Visual question answering
LIVE: Learnable In-Context Vector for Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIVE, a learnable per-layer in-context vector distilled from 32-shot demonstrations via KL alignment, which replaces ICL demonstrations in IDEFICS for VQA at a fraction of inference cost.
Problem
In-context learning in large multimodal models is expensive (many demonstration tokens) and sensitive to demonstration selection, and existing non-learnable in-context vector methods from NLP fail on complex VQA tasks.
Approach
Learn a per-layer shift vector and scalar weight that are trained to match the output distribution of a model prompted with random 32-shot demonstrations (KL distillation plus a ground-truth loss), so the vectors replace demonstrations at inference time.
Key results
- LIVE reaches 58.54% on VQAv2 and 50.08% on OKVQA versus 56.18% and 48.48% for 32-shot ICL on IDEFICS-9B, with 131K trainable parameters.
- LIVE uses about 1/24.97 the FLOPs and 1/8.25 the inference time of 32-shot ICL per forward pass.
- Non-learnable ICV baselines (TV, FV, PCA-ICV) fall well below 32-shot ICL on VQA; the best, TV, reaches 43.68% on VQAv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 32-shot demonstrations at inference with LIVE | IDEFICS-9B · VQAv2 | accuracy (%): 56.18 -> 58.54; +2.36 | Helped |
| Replace 32-shot demonstrations at inference with LIVE | IDEFICS-9B · OKVQA | accuracy (%): 48.48 -> 50.08; +1.6 | Helped |
| Use only L_gt instead of the combined loss (ablation) | LIVE · VQAv2 | accuracy (%): 58.54 -> 41.64; 16.9 lower | Hurt |
| Use only L_gt instead of the combined loss (ablation) | LIVE · OKVQA | accuracy (%): 50.08 -> 43.96; 6.12 lower | Hurt |
| Use only L_d instead of the combined loss (ablation) | LIVE · VQAv2 | accuracy (%): 58.54 -> 54.76; 3.78 lower | Hurt |
| Use only L_d instead of the combined loss (ablation) | LIVE · OKVQA | accuracy (%): 50.08 -> 46.94; 3.14 lower | Hurt |
| Increase the number of training demonstrations from 1 to 32 (ablation) | LIVE · VQAv2 | accuracy (%): 56.84 -> 58.54 | Helped |
| Increase the number of training demonstrations from 1 to 32 (ablation) | LIVE · OKVQA | accuracy (%): 47.51 -> 50.08 | Helped |
| Use one shared shift vector across all layers instead of layer-specific vectors (ablation) | LIVE · VQAv2 | accuracy (%): 58.54 -> 37.94; -20.6 | Hurt |
| Use one shared shift vector across all layers instead of layer-specific vectors (ablation) | LIVE · OKVQA | accuracy (%): 50.08 -> 41.07; -9.01 | Hurt |
| Train LIVE rather than use randomly initialized, untrained vectors (ablation) | LIVE · VQAv2 | accuracy (%): 30.3 -> 58.54 | Helped |
| Train LIVE rather than use randomly initialized, untrained vectors (ablation) | LIVE · OKVQA | accuracy (%): 30.2 -> 50.08 | Helped |
Surprising findings
- Using only the ground-truth loss (standard fine-tuning) degrades LIVE substantially, indicating the demonstration-distillation KL loss is essential.
- A single shared vector across all layers drops accuracy by 20.6 points on VQAv2, showing layer-specific vectors matter.
Limitations
- Evaluated on a single LMM family (IDEFICS) and mainly on VQA datasets; generalization is shown only in limited supplementary experiments.
- Non-learnable baselines were tuned over layers and hyperparameters on the evaluation data, and evaluation used a 10,000-question subset of VQAv2 validation due to compute limits.
Experimental setup
Datasets: VQAv2 OKVQA COCO captions OcrVQA VL-ICL
Benchmarks: VQAv2 OKVQA COCO captions OcrVQA VL-ICL
Metrics: accuracy CIDEr FLOPs runtime
Baselines: Zero-Shot 32-shot ICL Task Vector Function Vector PCA-ICV LoRA RICE MMICES
Builds on: Task Vector Function Vector PCA In-Context Vector LoRA
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.93)
- How to Configure Good In-Context Sequence for Visual Question Answering (CVPR 2024 · similarity 0.92)
- Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.89)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.89)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.