Research map / Large language models / Language, safety and interpretability
Link-Context Learning for Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes link-context learning (LCL), a causally-linked few-shot setting for MLLMs, fine-tunes Shikra on ImageNet-based contrastive support sets, and introduces the ISEKAI dataset of generated novel concepts for evaluation.
Problem
MLLMs fail to learn novel concepts and unseen images from in-context demonstrations, since standard ICL demonstrations are causally unrelated to the query and answers in demonstrations can be wrong without hurting performance.
Approach
Link-context learning requires the support set and query to be causally linked (e.g., a renamed concept in demonstrations must be used at inference), and trains MLLMs with class-prototype and hard-negative neighbor sampling from ImageNet so the model learns to use the support-query mapping.
Key results
- On ISEKAI-10 at 16-shot, Ours-[mix] reaches 0.62 accuracy vs 0.46 for OpenFlamingo and 0.07 for Otter.
- On ImageNet-100, Ours-[mix] reaches 83% accuracy at 6-shot, versus 78% for OpenFlamingo at its peak and 25% for Otter.
- Zero-shot VQAv2 accuracy of Ours-7B-[mix] is 75.06 (dev), close to Shikra-13B's 77.3, indicating little harm to zero-shot ability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train Shikra with link-context tasks mixed with its original tasks | Shikra · ImageNet-100, 6-shot | accuracy: 0.72 -> 0.83 | Helped |
| train with link-context tasks mixed with original tasks rather than Shikra alone | Shikra · ImageNet-100, zero-shot | accuracy: 0.05 -> 0.16 | Helped |
| train with link-context tasks mixed with original tasks rather than Shikra alone | Shikra · VQAv2dev, zero-shot | accuracy: 77.3 -> 75.06 | Hurt |
| train with link-context tasks mixed with original tasks rather than Shikra alone | Shikra · VQAv2std, zero-shot | accuracy: 77.5 -> 75.25 | Hurt |
| fine-tune across 2–16 shots using random shot sampling (ablation) | Shikra link-context model · ImageNet-100, 16-shot | accuracy: 0.79 -> 0.75 | Hurt |
| fine-tune across 2–16 shots using weighted shot sampling (ablation) | Shikra link-context model · ImageNet-100, 16-shot | accuracy: 0.79 -> 0.79 | No clear effect |
| use random-shot training rather than fixed-shot training (ablation) | Shikra link-context model · ImageNet-100, 2-shot | accuracy: 0.51 -> 0.77 | Helped |
| increase the false-label rate in the support set (ablation) | link-context model · ISEKAI | accuracy: 0.78 -> 0.00 | Hurt |
| increase the number of shots beyond 8 (ablation) | link-context model · ImageNet-100 | accuracy: 0.83 at 6-shot; 0.57 at 16-shot | Mixed |
| modify labels at different positions in the 16-shot context (ablation) | link-context model · ImageNet-100 | accuracy: 0.63; 0.69; 0.63; 0.68; 0.7 | Mixed |
Surprising findings
- Accuracy drops to 0% when the labels in the support set are all false, while OpenFlamingo keeps about 38% accuracy at a 100% false rate, suggesting it ignores the support mapping.
- Performance of the model depends on label position in the support set, with strong reliance on the beginning and middle positions (lost-in-the-middle effect).
Limitations
- Evaluation focuses on basic image classification; the authors describe the work as a foundational baseline.
- ISEKAI contains only 20 groups and 40 categories at the time of writing, a small evaluation set.
- Tuning on ImageNet-derived classes may limit generalization beyond classification.
Experimental setup
Datasets: ImageNet-1k ISEKAI
Benchmarks: ISEKAI-10 ISEKAI-pair ImageNet-100 VQAv2
Metrics: accuracy
Baselines: OpenFlamingo Otter Shikra-13B Flamingo BLIP2
Builds on: Shikra in-context learning CLIP Flamingo
Closest papers by meaning
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.90)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.90)
- Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning (NeurIPS 2024 · similarity 0.89)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.89)
- ContextNav: Towards Agentic Multimodal In-Context Learning (ICLR 2026 · similarity 0.88)
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.