Research map / Vision-language and multimodal / Multimodal LLMs
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRESS, an LVLM trained with GPT-4-generated natural language feedback (critique and refinement NLF) via generalized conditional RL, improving helpfulness, honesty, harmlessness and multi-turn refinement over prior LVLMs.
Problem
Existing LVLMs rely mainly on SFT for human alignment, producing unhelpful, hallucinated or harmful responses, and their multi-turn instruction data has weak inter-turn dependencies, limiting interaction ability.
Approach
Categorize LLM-generated natural language feedback into critique NLF (strengths/weaknesses, for alignment) and refinement NLF (concrete suggestions, for multi-turn interaction), and generalize conditional RL by training the LVLM with LM loss to generate responses conditioned on numerical scores and NLF, using the best-score prefix at inference.
Key results
- DRESS achieves relative improvements of 9.76% helpfulness, 11.52% honesty, and 21.03% harmlessness over prior SOTA LVLMs.
- On VLSafe, DRESS reaches a safety score of 88.56 versus 75.05 for MiniGPT-4 (next best).
- On COCO captioning, DRESS reduces CHAIRs under Instruction-2 to 9.84 from 14.48 for InstructBLIP and 73.50 for mPLUG-Owl.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply natural-language feedback training (RLAIF) (ablation) | DRESS · LLaVA Eval Average | GPT-4 score: 70.05 -> 74.42 | Helped |
| Add critique NLF alongside numerical scores (ablation) | DRESS · LLaVA Eval Average | GPT-4 score: 71.59 -> 74.42 | Helped |
| Add refinement NLF and multi-turn training (ablation) | DRESS · LLaVA Eval Average | GPT-4 score: 73.47 -> 74.42 | Helped |
| Add honesty feedback (ablation) | DRESS · LLaVA Eval Average | GPT-4 score: 73.88 -> 74.42 | Helped |
| Add helpfulness feedback (ablation) | DRESS · LLaVA Eval Average | GPT-4 score: 73.23 -> 74.42 | Helped |
| Use DRESS instead of InstructBLIP | LVLMs · LLaVA Eval Conversation | GPT-4 score: 74.08 -> 77.67 | Helped |
| Use DRESS instead of LLaVA-HF | LVLMs · LLaVA Eval Reasoning | GPT-4 score: 85.33 -> 84.27 | Hurt |
| Use DRESS instead of InstructBLIP | LVLMs · LLaVA Eval Average | GPT-4 score: 72.64 -> 74.70 | Helped |
| Use DRESS instead of miniGPT4 | LVLMs · LLaVA Bench | Helpfulness: 24.17 -> 26.45 | Helped |
| Use DRESS instead of BLIP-2 | LVLMs · COCO image captioning, Instruction-2 | CHAIRi: 2.75 -> 4.74 | Hurt |
| Use DRESS instead of BLIP-2 | LVLMs · COCO image captioning, Instruction-2 | CHAIRs: 3.50 -> 9.84 | Hurt |
| Use DRESS instead of miniGPT4 | LVLMs · VLSafe test | Safety: 75.05 -> 88.56 | Helped |
Surprising findings
- Instruction finetuning appears to increase hallucinated objects: BLIP-2 without instruction tuning has lower CHAIR scores than instruction-tuned LVLMs.
- Using external feedback for alignment does not improve fundamental visual accuracy; DRESS is comparable on the Accuracy dimension.
Limitations
- Feedback relies on GPT-4 rather than human annotation; human agreement is only checked in appendix.
- Harmlessness evaluation is limited to harm definitions within GPT-4.
- External feedback does not improve fundamental visual accuracy.
Experimental setup
Datasets: LLaVA visual instruction tuning data CC3M CC12M SBU COCO VLSafe
Benchmarks: LLaVA-Eval LLaVA-Bench COCO image captioning (CHAIR) VLSafe VQAv2 OK-VQA GQA
Metrics: GPT-4 helpfulness score CHAIRi CHAIRs safety score persuasiveness VQA accuracy (GPT-3.5 judged)
Baselines: BLIP-2 InstructBLIP LLaVA LLaVA-HF mPLUG-Owl MiniGPT-4
Builds on: LLaVA conditional reinforcement learning RLAIF CHAIR
Closest papers by meaning
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.89)
- ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.88)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.88)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.