Research map / Vision-language and multimodal / Multimodal LLMs
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FIRE, a 1.1M-conversation feedback-refinement dataset (GPT-4V seed plus student/teacher self-generated data), a 11K-conversation benchmark, and FIRE-LLaVA, a LLaVA-NeXT-8B fine-tuned to refine responses from feedback.
Problem
Vision-language models often produce flawed responses and do not use user feedback to refine them; existing data for learning this feedback-refining behavior is small and narrow in task coverage.
Approach
Generate multi-turn student-teacher feedback-refinement dialogues: GPT-4V seeds 100K conversations, then LoRA-tuned LLaVA-NeXT student and teacher models self-generate ~1M more, evaluated by a new benchmark with fixed and free dialogue settings and metrics (AT, ADR, ATR, RR).
Key results
- FIRE-LLaVA improves over LLaVA-NeXT-8B in free dialogues: ADR 1.56 vs 0.97, ATR 0.66 vs 0.41, RR 0.39 vs 0.25, and AT 0.84 vs 1.00.
- On instruction-following benchmarks FIRE-LLaVA is comparable to LLaVA-NeXT-8B (e.g., VQAv2 82.9 vs 79.0, MMBench 78.5 vs 74.4, MME-P 1534.8 vs 1546.0).
- In human preference evaluation on 100 instances, FIRE-LLaVA was preferred over LLaVA-NeXT-8B (37.67 vs 24.33 votes, 38 ties).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA-NeXT-8B on FIRE-100K and FIRE-1M | LLaVA-NeXT-8B student model · FIRE-Bench fixed dialogues | BLEU-1: 0.33 -> 0.54 | Helped |
| Fine-tune LLaVA-NeXT-8B on FIRE-100K and FIRE-1M | LLaVA-NeXT-8B student model · FIRE-Bench fixed dialogues | CIDEr: 0.60 -> 2.36 | Helped |
| Train a teacher model on FIRE-100K | LLaVA-NeXT-8B · FIRE-Bench fixed dialogues | MAE: 1.88 -> 0.30 | Helped |
| Fine-tune on FIRE-100K and FIRE-1M | LLaVA-NeXT-8B student model · FIRE-Bench free dialogues overall | ADR: 0.97 -> 1.56 | Helped |
| Fine-tune on FIRE-100K and FIRE-1M | LLaVA-NeXT-8B student model · FIRE-Bench free dialogues overall | AT: 1 -> 0.84 | Helped |
| Add FIRE-1M to training data (ablation) | FIRE100K-LLaVA · FIRE-Bench free dialogues overall | ADR: 1.27 -> 1.56 | Helped |
| Add FIRE-1M to training data (ablation) | FIRE100K-LLaVA · FIRE-Bench free dialogues overall | AT: 0.92 -> 0.84 | Helped |
| Increase FIRE training data, combining FIRE-100K with more FIRE-1M data (ablation) | LLaVA-NeXT-8B · FIRE-Bench | feedback-refinement performance | Helped |
| Use FIRE-trained LLaVA rather than the original model | LLaVA-NeXT-8B · FIRE-Bench free dialogues, five turns | correct-response rate: 49.60% -> 69.19% vs. 46.57% -> 59.37% | Helped |
| Replace LLaMA3-8B with Vicuna1.5-7B and fine-tune on FIRE (ablation) | LLaVA-Next-Vicuna · FIRE-Bench free dialogues overall | ADR: 0.98 -> 1.11 | Helped |
| Fine-tune LLaVA-1.5-Vicuna-7B on FIRE-100K | LLaVA-1.5-Vicuna-7B · FIRE-Bench | RR: 0.12 -> 0.20 | Helped |
Surprising findings
- Human scoring of FIRE-100K and FIRE-1M data quality is high (feedback ~4.8/5), suggesting student-teacher self-generated data stays close to GPT-4V-generated quality.
Limitations
- Feedback is limited to text; no multimodal feedback such as pointing to image regions.
- Quality filtering cannot fully guarantee data quality.
- Relies on GPT-4V for the seed data and on GPT-4V as judge for evaluation.
Experimental setup
Datasets: FIRE-100K FIRE-1M LLaVA-665K VQAv2 GQA TextVQA ChartQA DocVQA DVQA GeoQA+ Synthdog-EN MathVista
Benchmarks: FIRE-Bench GQA VQAv2 VizWiz TextVQA ScienceQA-IMG LLaVA-Bench-in-the-wild MMBench MME MM-Vet
Metrics: BLEU CIDEr MAE average turn average dialogue refinement average turn refinement refinement ratio
Baselines: LLaVA-NeXT-8B LLaVA-1.5-Vicuna-7B LLaVA-NeXT-Vicuna-7B DRESS
Builds on: LLaVA-NeXT LoRA Self-Refine DRESS
Closest papers by meaning
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences (NeurIPS 2024 · similarity 0.87)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.87)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.87)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.87)
- DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · similarity 0.87)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.