Research map / Vision-language and multimodal / Multimodal LLMs
Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SENA, a self-evolution framework that lets an MLLM generate its own questions and preference answers from unlabeled images, using image-driven self-questioning, answer self-enhancement, and an image content alignment loss added to DPO.
Problem
Multimodal preference alignment via self-evolution still depends on human- or GPT-annotated questions, extra reward/teacher models, or ground-truth answers, which raises cost and complexity.
Approach
A single model generates questions from unlabeled images, self-checks answerability and regenerates unanswerable ones, improves chosen answers using a self-generated image description, creates rejected answers from diffusion-noised images, and adds a description likelihood loss (CA) to DPO to reduce hallucination.
Key results
- SENA (LLaVA-1.5-7B, 3 iterations) reaches LLaVA-Bench 67.4 vs 59.6 for the base model and MMHal score 2.33 vs 1.90, with hallucination rate 0.52 vs 0.61.
- On AMBER discriminative, SENA achieves 79.4 accuracy and 83.6 F1 vs 71.7 and 74.3 for the base model, matching or exceeding annotation-based methods such as SeVa on that task.
- SENA attains MMHal rate 0.52 without external models, while RLAIF-V (which uses an extra 34B model) achieves 0.34 on MMHal and lower CHAIR, so SENA trails it on hallucination metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image-driven self-questioning (SQ) (ablation) | LLaVA-1.5-7B self-evolution baseline · LLaVA-Bench / AMBER / MMBench | LLaVA-Bench score: 62.8 -> 64.9 | Helped |
| add answer self-enhancement (SE) (ablation) | LLaVA-1.5-7B self-evolution baseline · LLaVA-Bench / AMBER | LLaVA-Bench score: 62.8 -> 65.0 | Helped |
| add image content alignment loss (CA) (ablation) | LLaVA-1.5-7B self-evolution baseline · LLaVA-Bench / AMBER-Dis. | LLaVA-Bench score and AMBER-Dis F1: LLaVA-Bench 62.8 -> 66.7; AMBER-Dis F1 72.4 -> 77.9 | Helped |
| add SQ, SE and CA together, iterated 3 times (ablation) | LLaVA-1.5-7B · LLaVA-Bench / MMHal / AMBER | LLaVA-Bench score, MMHal rate, AMBER-Dis F1: LLaVA-Bench 59.6 -> 67.4; MMHal rate 0.61 -> 0.52; AMBER-Dis F1 74.3 -> 83.6 | Helped |
| iterate self-evolution beyond 3 rounds (plateau after round 3) | LLaVA-1.5-7B SENA · MMBench | accuracy: 65.0 (θ1) -> 65.2 (θ2) -> 64.8 (θ3) | No clear effect |
| apply SE to descriptive answer only vs. generated answer only vs both (ablation) | LLaVA-1.5-7B · LLaVA-Bench / AMBER | LLaVA-Bench score: 62.8 -> 63.9 (y_des only), 63.7 (y_gen only), 65.0 (both) | Helped |
| apply alignment loss to descriptive answers only vs generated answers only (ablation) | LLaVA-1.5-7B · LLaVA-Bench / AMBER | AMBER CHAIR and LLaVA-Bench: LLaVA-Bench 66.7 (y_des only) vs 63.3 (y_gen only); CHAIR 6.5 vs 6.5 | Mixed |
| increase diffusion noise steps T from 200 to 600 for rejected answers (ablation) | LLaVA-1.5-7B SENA · LLaVA-Bench / AMBER | LLaVA-Bench score: 64.2 (T=200) -> 66.9 (T=600) | Helped |
| use descriptive and generated questions combined vs single type (ablation) | LLaVA-1.5-7B SENA · LLaVA-Bench / AMBER | LLaVA-Bench score: 64.5 (q_gen only) / 62.7 (q_des only) -> 65.3 (both) | Helped |
| scale number of images per iteration M from 2K to 6K (beyond 6K plateaus) (ablation) | LLaVA-1.5-7B SENA · LLaVA-Bench | LLaVA-Bench score: 63.6 (2K) -> 64.8 (4K) -> 66.9 (6K) -> 67.3 (8K) | Helped |
| apply SENA to LLaVA-1.5-13B | LLaVA-1.5-13B · LLaVA-Bench | score: base 66.8 -> 71.8 after 3 iterations | Helped |
| apply SENA to Qwen2-VL-7B | Qwen2-VL-7B · LLaVA-Bench | score: base 86.6 -> 89.2 after 2 iterations | Helped |
Surprising findings
- Performance plateaus after three evolution rounds, and increasing image count beyond 6K gives no further gain.
- Using the GPT-4 judge with different API versions changes scores substantially (e.g., GPT-4o gives lower LLaVA-Bench scores than GPT-4-1106-preview), so evaluation results are sensitive to judge version.
Limitations
- Performance on some benchmarks lags behind self-evolution methods that use annotated data (e.g., RLAIF-V on hallucination metrics).
- Performance plateaus after three rounds of evolution.
- Diversity of self-generated questions is not fully addressed; the authors note hallucinated samples increase with more images.
Experimental setup
Datasets: LLaVA-665K images (COCO, GQA, TextVQA, OCRVQA, Visual Genome)
Benchmarks: LLaVA-Bench (In-the-Wild) MM-Vet MMHal-Bench AMBER MMBench GQA SQA TextVQA VQAv2
Metrics: GPT-4 score ratio MMHal score hallucination rate CHAIR Cover Hal Cog accuracy F1 CircularEval accuracy
Baselines: LLaVA-1.5-7B SeVa STIC SIMA RLAIF-V CSR BLIP-2 MiniGPT-4 InstructBLIP-7B Shikra-13B Qwen-VL-7B mPLUG-Owl2
Builds on: DPO iterative DPO LLaVA-1.5 STIC SeVa RLAIF-V SIMA CSR
Closest papers by meaning
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.89)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.87)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.87)
- Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.