Research map / Vision-language and multimodal / CLIP and zero-shot
Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Image-Conditioned Caption Correction (ICCC), a second-stage tuning task that builds caption-correction samples from unlabeled COCO/VG captions using a spaCy dependency parser, improving zero-shot VQA and captioning of BLIP-2 and InstructBLIP without human or LLM labels.
Problem
Zero-shot reasoning of generative VLMs usually needs second-stage instruction tuning with human-labeled or LLM-generated task data, which is costly.
Approach
Construct concept-mismatched captions by replacing or swapping dependency-parsed linguistic units (entities, predicates, attributes, nouns, verbs) in unlabeled image-text captions, and train the VLM to correct them conditioned on the image using the standard language modeling objective.
Key results
- BLIP-2 OPT2.7B with ICCC improves GQA from 33.5 to 38.9 and NoCaps CIDEr from 105.7 to 111.9 zero-shot.
- BLIP-2 OPT6.7B with ICCC improves VQAv2 from 52.6 to 58.8.
- InstructBLIP with ICCC improves GQA from 48.4 to 49.8, VSR from 61.1 to 63.1, and NoCaps SPICE from 14.2 to 15.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICCC tuning | BLIP-2 OPT2.7B · GQA testdev | accuracy (%): 33.5 -> 38.9 | Helped |
| Add ICCC tuning | BLIP-2 OPT2.7B · VSR testdev | accuracy (%): 48.3 -> 47.8 | Hurt |
| Add ICCC tuning | BLIP-2 OPT2.7B · NoCaps validation | CIDEr: 105.7 -> 111.9 | Helped |
| Add ICCC tuning | InstructBLIP · VSR | accuracy (%): 61.1 -> 63.1 | Helped |
| Add ICCC samples to COCO/VG caption training (ablation) | BLIP-2 OPT2.7B · GQA | accuracy (%): 33.0 -> 38.9 | Helped |
| Add VQAv2 training data to COCO/VG caption training (ablation) | BLIP-2 OPT2.7B · GQA | accuracy (%): 33.0 -> 41.3 | Helped |
| Add ICCC samples to COCO/VG and VQAv2 training (ablation) | BLIP-2 OPT2.7B · GQA | accuracy (%): 41.3 -> 42.8 | Helped |
| Use noun and verb concepts only (ablation) | ICCC on BLIP-2 OPT2.7B · NoCaps validation | CIDEr: 108.1 -> 114.3 | Helped |
| Use entity, predicate, and attribute concepts (ablation) | ICCC on BLIP-2 OPT2.7B · GQA | accuracy (%): 33.0 -> 38.3 | Helped |
| Use noun and entity concepts only (ablation) | ICCC on BLIP-2 OPT2.7B · NoCaps validation | BLEU@4: 45.6 -> 46.5 | Helped |
| Use verb, predicate, and attribute concepts (ablation) | ICCC on BLIP-2 OPT2.7B · NoCaps validation | CIDEr: 108.1 -> 114.1 | Helped |
| Randomly alter words instead of using language structure (ablation) | ICCC on BLIP-2 OPT2.7B · VQAv2 | accuracy (%): 49.7 -> 48.6 | Hurt |
Surprising findings
- Adding correction instructions and random word masking without language structure did not improve downstream generalization, highlighting the importance of structure.
- BLIP-2 FlanT5-XL tends to overfit on ICCC samples, requiring a very small proportion (pc=0.01) of them.
Limitations
- Due to limited computing resources, extension to larger datasets and models was not explored.
- Hyperparameters pc and ps must be tuned, and excessive swap-based correction samples cause language bias.
- Results are limited to BLIP-2 and InstructBLIP variants with 2.7B to 7B LLMs.
Experimental setup
Datasets: COCO Caption Visual Genome Caption
Benchmarks: GQA OK-VQA VQAv2 VSR NoCaps
Metrics: top-1 accuracy BLEU@4 SPICE CIDEr
Baselines: BLIP-2 InstructBLIP
Builds on: BLIP-2 InstructBLIP NegCLIP spaCy
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.89)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.