Research map / Vision-language and multimodal / CLIP and zero-shot
Large (Vision) Language Models are Unsupervised In-Context Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes joint inference over multiple unlabeled inputs, approximated by unsupervised fine-tuning (LoRA task encoder trained with a REINFORCE-style estimator) and unsupervised ICL (iterative self-labeled refinement), evaluated on Llama-3.1-8B, Qwen2.5-Math-7B, OpenFlamingo, and GPT-4o.
Problem
Zero-shot inference makes independent predictions and existing adaptation (ICL, fine-tuning, prompt engineering) needs labeled examples or manual prompt effort; the paper asks how to improve a foundation model on a task without any labels.
Approach
Generalize zero-shot inference to joint prediction over N inputs, maximizing the model's joint likelihood of its own predictions; amortize this with a LoRA task encoder trained via a variance-reduced REINFORCE estimator (unsupervised FT), and approximate it with iterative in-context relabeling using self-generated labels (unsupervised ICL) that works for closed models.
Key results
- Unsupervised fine-tuning on Llama-3.1-8B improves zero-shot average over 13 NLP tasks from 60.1% to 83.1%, close to fully supervised LoRA (90.1%).
- Unsupervised ICL on Qwen2.5-Math-7B raises GSM8K accuracy from 52.2% to 91.4%, above supervised ICL (89.9%).
- Unsupervised ICL on GPT-4o improves ImageNet-100 from 76.1% (zero-shot CoT) to 79.0%, versus 79.5% for supervised ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply unsupervised fine-tuning with LoRA instead of zero-shot inference | Llama-3.1-8B · NLP average (13 datasets) | accuracy: 60.1 -> 83.1 | Helped |
| Apply unsupervised ICL instead of zero-shot inference | Llama-3.1-8B · NLP average (13 datasets) | accuracy: 60.1 -> 79.3 | Helped |
| Apply unsupervised ICL with CoT instead of zero-shot | Qwen2.5-Math-7B · GSM8K | accuracy: 52.2 -> 91.4 | Helped |
| Apply unsupervised ICL instead of zero-shot | OpenFlamingo-4B · Food101 | accuracy: 58.4 -> 81.0 | Helped |
| Apply unsupervised ICL with CoT instead of zero-shot CoT | GPT-4o · ImageNet-100 | accuracy: 76.1 -> 79.0 | Helped |
| Apply unsupervised ICL with CoT instead of zero-shot CoT | GPT-4o · MMMU | accuracy: 66.4 -> 68.6 | Helped |
| Apply unsupervised ICL with CoT instead of zero-shot CoT | GPT-4o · MMMU-Pro | accuracy: 54.7 -> 55.5 | Helped |
| Increase context length N (ablation) | Unsupervised fine-tuning and unsupervised ICL · Text and image classification tasks | accuracy | Helped |
| Replace LoRA with a linear task encoder (ablation) | Unsupervised fine-tuning · SST2, RTE, BoolQ | accuracy: 3%, 9% and 3% | Hurt |
| Use full-model fine-tuning instead of LoRA (ablation) | Unsupervised fine-tuning · RTE | accuracy: 7% | Hurt |
Surprising findings
- Unsupervised ICL on GSM8K with Qwen2.5-Math-7B slightly outperforms supervised ICL, which the authors attribute to refinement of reasoning chains.
- Unsupervised fine-tuning with full-model training overfits on RTE and performs worse than LoRA-based unsupervised fine-tuning.
Limitations
- Unsupervised fine-tuning is limited to close-ended tasks with a finite answer set and requires model weights and output probabilities.
- The framework depends on the base model having in-context learning ability; gains are small when ICL is weak.
- Unsupervised ICL generally falls short of unsupervised fine-tuning on NLP tasks.
Experimental setup
Datasets: SST2 Amazon AG-News TREC DBpedia-14 SUBJ RTE QNLI MNLI COPA BoolQ PIQA
Benchmarks: SST2 Amazon AG-News TREC DBpedia-14 SUBJ RTE QNLI MNLI COPA BoolQ PIQA
Metrics: accuracy
Baselines: zero-shot inference zero-shot chain-of-thought supervised ICL fully-supervised LoRA fine-tuning zero-shot with instruction-tuned Llama-3.1-8B-Instruct
Builds on: chain-of-thought LoRA Gibbs sampling REINFORCE
Closest papers by meaning
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.88)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.88)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.88)
- Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.