Research map / Vision-language and multimodal / Multimodal LLMs
Visual Instruction Tuning with Polite Flamingo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Polite Flamingo, a multimodal response rewriter trained to restore polished responses from distorted annotations, uses it to build the PF-1M dataset, and fine-tunes Clever Flamingo with U-shaped multi-stage tuning to reduce the "multi-modal alignment tax" on response politeness.
Problem
Visual instruction tuning on raw, terse vision-language dataset annotations teaches multimodal LLMs unformatted, impolite response styles, reducing human preference (a "multi-modal alignment tax").
Approach
Train a multimodal rewriter (OpenFlamingo-9B with LoRA) to reconstruct high-quality GPT-4/ChatGPT-style responses from automatically distorted versions, then apply it to rewrite raw annotations of 37 vision-language datasets into the 0.97M-sample PF-1M dataset for visual instruction tuning, with U-shaped three-stage tuning and multi-turn augmentation.
Key results
- Polite Flamingo raises average reward score of raw annotations from -2.42 to -0.50 on instruction datasets, and PF-1M outperforms M3IT in reward.
- Clever Flamingo beats Otter on all captioning and VQA benchmarks, e.g. COCO Rouge-L 24.3 vs 22.6 and OK-VQA 43.3 vs 28.7 (zero-shot).
- Clever Flamingo has a 62.1% win rate against Otter by reward model and beats all compared models except LLaVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Polite Flamingo rewriting of raw annotations into PF-1M used for visual instruction tuning (vs raw annotations) (ablation) | Clever Flamingo · Ablation Figure (held-in/held-out accuracy and reward) | politeness (reward score) | Helped |
| Raw annotations instead of Polite Flamingo rewriting (ablation) | Clever Flamingo · Held-in/held-out VQA accuracy | NLI-based accuracy | Mixed |
| Stage 3 low-LR LoRA tuning on politeness after stage 2 (ablation) | Clever Flamingo U-shaped tuning · Ablation Figure | politeness (reward score) | Helped |
| Skipping stage 1 (direct stage 2 from OpenFlamingo-9B) (ablation) | Clever Flamingo · Ablation Figure | OOD VQA accuracy | Hurt |
| Image resolution 224x224 instead of 336x336 in stage 2 (ablation) | Clever Flamingo · Ablation Figure | VQA accuracy | Hurt |
| Unfreezing ViT during stage 2 (ablation) | Clever Flamingo · Ablation Figure | VQA accuracy | No clear effect |
| Doubling stage 2 epochs from 3 to 6 (ablation) | Clever Flamingo · Ablation Figure | OOD VQA accuracy | Hurt |
| Unfreezing LoRA during stage 2 (ablation) | Clever Flamingo · Ablation Figure | held-in accuracy / OOD accuracy | Mixed |
| Second-generation Polite Flamingo (initialized from Clever Flamingo) vs first generation (ablation) | Polite Flamingo rewriter · A-OKVQA rewriting | average reward: +0.51 | Helped |
| Second-generation Polite Flamingo (initialized from Clever Flamingo) vs first generation (ablation) | Polite Flamingo rewriter · CC-3M rewriting | CLIPScore | Hurt |
| Distorted-response training of Polite Flamingo (reconstruct polite responses from distorted ones) to rewrite raw annotations, then tune Clever Flamingo on PF-1M | Clever Flamingo vs Otter (same base OpenFlamingo-9B) · COCO / TextCaps / OK-VQA / Grid-3D | Rouge-L / NLI accuracy: +1.7 COCO, +4.4 TextCaps, +14.6 OK-VQA, +15.5 Grid-3D | Helped |
| Clever Flamingo tuning on PF-1M vs Otter | Multi-image reasoning · Spot-the-Diff, Image-editing-requests, NLVR2 | STS / Rouge-L: +6.6 STS Spot-the-Diff, +3.9 STS Image-editing, +16.7 STS NLVR2 | Helped |
Surprising findings
- Training directly on raw annotations slightly improved held-in and held-out accuracy but significantly reduced politeness, illustrating the alignment tax trade-off.
- A second-generation Polite Flamingo, initialized from Clever Flamingo, improved reward score but did not improve CLIPScore on CC-3M captions.
Limitations
- Rewriter is based on a 7B model and sometimes changes answers or introduces hallucinations, requiring filtering.
- Clever Flamingo still exhibits severe object hallucination.
- Comparisons with InstructBLIP and others are not fully aligned in base model and data scale.
Experimental setup
Datasets: LLaVA-instruct-177k UltraChat ShareGPT MS-COCO VQA-v2 GQA A-OKVQA OCR-VQA TextCaps Flickr-30k CC-3M ScienceQA
Benchmarks: MS-COCO TextCaps Image2Paragraph OK-VQA VSR Grid-3D Spot-the-Diff Image-editing-requests NLVR2
Metrics: Rouge-L NLI-based VQA accuracy STS reward model score win rate CLIPScore
Baselines: MiniGPT-4 LLaVA InstructBLIP Otter M3IT MIMIC-IT
Builds on: OpenFlamingo LLaVA QLoRA Flamingo
Closest papers by meaning
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.84)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.84)
- Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.84)
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates (NeurIPS 2024 · similarity 0.83)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.83)
- DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.