Research map / Vision-language and multimodal / CLIP and zero-shot
Test-Time Visual In-Context Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes test-time Visual In-Context Tuning (VICT), which adapts the Painter VICL model per test sample using a flipped-prompt cycle-consistency regression loss, improving robustness under 15 corruptions across six dense vision tasks.
Problem
Visual in-context learning models like Painter generalize poorly under distribution shifts (image corruptions), even when given in-context examples from the test distribution.
Approach
Flip the roles of task prompt and test sample in the four-cell grid canvas: predict the test output, use it as a prompt to reconstruct the original task prompt output, and optimize the encoder with a regression loss on this cycle-consistency signal, resetting weights per test sample.
Key results
- On ADE20K-C semantic segmentation, one-shot VICT reaches 29.3 mIoU average versus 25.8 for one-shot Painter across 15 corruptions at level 5.
- On NYUv2-C depth estimation, one-shot VICT reduces average A.Rel to 0.205 from 0.537 for Painter one-shot.
- Zero-shot VICT outperforms one-shot Painter across tasks, and VICT nearly matches or beats 64-shot fine-tuned Painter on ADE20K-C and COCO-C.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply test-time cycle-consistency tuning | Painter, zero-shot · NYUv2-C | A.Rel: 0.392 -> 0.365 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · NYUv2-C | A.Rel: 0.537 -> 0.205 | Helped |
| Apply test-time cycle-consistency tuning | Painter, zero-shot · ADE20K-C | mIoU: 27.0 -> 28.2 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · ADE20K-C | mIoU: 25.8 -> 29.3 | Helped |
| Apply test-time cycle-consistency tuning | Painter, zero-shot · COCO-C | PQ: 26.4 -> 26.9 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · COCO-C | PQ: 25.3 -> 26.8 | Helped |
| Apply test-time cycle-consistency tuning | Painter, zero-shot · SIDD-C | PSNR: 23.08 -> 23.22 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · SIDD-C | PSNR: 23.12 -> 24.56 | Helped |
| Apply test-time cycle-consistency tuning | Painter, zero-shot · Rain-C | PSNR: 17.79 -> 17.97 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · Rain-C | PSNR: 17.77 -> 19.92 | Helped |
| Apply test-time cycle-consistency tuning | Painter, zero-shot · LoL-C | PSNR: 15.90 -> 16.15 | Helped |
| Apply test-time cycle-consistency tuning | Painter, one-shot · LoL-C | PSNR: 15.93 -> 16.94 | Helped |
Surprising findings
- Giving Painter corrupted one-shot task prompts from the test distribution often degrades performance versus clean zero-shot prompts.
- Optimizing only the encoder at test time performs similarly to optimizing encoder and decoder.
Limitations
- Per-sample test-time optimization makes inference slower than a frozen model.
- Cycle-consistency signal may not produce useful gradients for every test distribution.
- Evaluation limited to a few dense tasks and corruption benchmarks.
Experimental setup
Datasets: NYUv2 ADE20K COCO SIDD Merged deraining datasets LoL
Benchmarks: NYUv2-C ADE20K-C COCO-C SIDD-C Rain-C LoL-C
Metrics: A.Rel mIoU PQ PSNR
Baselines: Painter Painter few-shot fine-tuned
Builds on: Painter MAE-VQGAN test-time training MAE
Closest papers by meaning
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.92)
- Towards Global Optimal Visual In-Context Learning Prompt Selection (NeurIPS 2024 · similarity 0.90)
- Towards Reliable and Holistic Visual In-Context Learning Prompt Selection (NeurIPS 2025 · similarity 0.89)
- Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning (CVPR 2025 · similarity 0.88)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.88)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.