Research map / Vision-language and multimodal / Multimodal LLMs
WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WEAVE, a dataset (WEAVE-100k) of 100K multi-turn interleaved editing/comprehension/generation samples and a human-annotated benchmark (WEAVEBench) with a hybrid VLM-judge, showing fine-tuning Bagel on the data improves comprehension, editing, and visual memory.
Problem
Existing datasets and benchmarks for unified multimodal models focus on single-turn interactions and do not capture multi-turn, context-dependent image creation and editing requiring visual memory.
Approach
A large-scale interleaved multi-turn dataset built from four generation pathways (multi-image fusion, remove-then-back, derivative imagination/comparison, sequential procedures) plus a benchmark scored by a hybrid VLM judge that uses both reference images and original-image-plus-instruction context.
Key results
- Fine-tuning Bagel on WEAVE-100k raises WEAVEBench average from 0.446 to 0.640 (about 42.5% relative gain) and gives a 9.8% gain on MMMU.
- On GEdit-Bench, fine-tuned Bagel reaches 6.83 average vs 6.52 for base Bagel (about 4.8% gain), with notable material and style gains.
- Best WEAVEBench in-context UMM result is Nano Banana at 0.767 average; open-source Qwen-Image-Edit and Bagel degrade with more in-context content.
Surprising findings
- Open-source editing models such as Qwen-Image-Edit degrade as in-context history increases, while proprietary Nano Banana improves.
- Concatenating sequential images hurts Bagel by 10.3%, showing sequential multi-image input matters.
Limitations
- Judge reliance on GPT-4.1 scores, validated only on 100 instances per model with three human experts.
- Heavy reliance on closed-source models (GPT-4.1, Seedream 4.0, Nano Banana) for data generation and filtering.
- Long multi-image contexts exceed GPU limits, so training samples single turns randomly rather than full dialogues.
Experimental setup
Datasets: WEAVE-100k Echo-4o multi-image fusion data Zebra-CoT PhysBench MMMU GEdit-Bench RISEBench GenEval MMBench MMVet
Benchmarks: WEAVEBench GEdit-Bench RISEBench MMMU MMBench MMVet GenEval
Metrics: Key Point Correctness Visual Consistency Image Quality Accuracy composite weighted score
Baselines: Bagel Bagel-Zebra Qwen-Image-Edit FLUX.1 Kontext Step1X-Edit OmniGen2 Nano Banana Seedream 4.0 GPT-4o GPT-4.1 Qwen3-VL InternVL3.5
Builds on: Bagel Echo-4o Zebra-CoT GEdit-Bench RISEBench
Closest papers by meaning
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.92)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.91)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.91)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.