Research map / Vision-language and multimodal / Human and embodied modeling

Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models

NeurIPS 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Scene graph generation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a tiny two-layer Transformer (Lever-LM) to select and order in-context demonstrations for frozen LVLMs (OpenFlamingo, IDEFICS), improving VQA and captioning ICL over retrieval baselines.

Problem

In-context learning performance of large vision-language models is highly sensitive to which demonstrations are selected and in what order, and existing ICD configuration methods from NLP rely on token probabilities that do not apply to continuous image patches.

Approach

Treat ICD sequence configuration as language modeling over a vocabulary of supporting-set examples: build a dataset of high-quality ICD sequences scored by the target LVLM's confidence in the ground-truth answer, then train a tiny Transformer on it to jointly select and order demonstrations conditioned on the query.

Key results

• On VQA with IDEFICS, Lever-LM reaches 48.75 vs 45.68 for the best baseline (STTR) in Avg:1~2 accuracy.
• On IC with OpenFlamingo, Lever-LM reaches 84.32 vs 78.29 CIDEr (SIIR) in Avg:1~2, and 92.45 Avg:1~8 vs 89.91 for SIIR.
• Lever-LM trained on 2-shot ICD data extrapolates to 3-8 shots, with Avg:3~8 CIDEr 105.79 vs 102.86 for RS on IDEFICS.

Measured effects reported in the paper

Surprising findings

• Training Lever-LM with 4-shot ICD sequences performed worse than with 2-shot sequences (IC Avg:1~8 about 1.57 lower), attributed to sub-optimal construction.
• Lever-LM sizes of 1-4 Transformer layers made minimal difference in captioning CIDEr.

Limitations

• The dataset construction strategy is sub-optimal, as shown by the 4-shot Lever-LM underperforming the 2-shot version.
• Requires ground-truth labels and a forward pass of the frozen LVLM to build training data, so it cannot be used at test time for label-free construction.
• Results on the golden fixed ICD sequence fluctuate significantly across runs, with one version underperforming random retrieval on VQA.

Experimental setup

Datasets: MS-COCO 2017 VQAv2

Benchmarks: MS-COCO VQAv2 VL-ICL CLEVR VL-ICL OCRText SST-2

Metrics: CIDEr accuracy

Baselines: Random Sample SIIR SITR STTR

Builds on: Flamingo CLIP MI-based prompt selection Retrieval-based ICD selection

Closest papers by meaning

• How to Configure Good In-Context Sequence for Visual Question Answering (CVPR 2024 · similarity 0.89)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
• LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.88)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.88)
• Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.87)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)