Research map / Vision-language and multimodal / Multimodal LLMs

Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Data-Juicer Sandbox, an open-source feedback-driven suite and "Probe-Analyze-Refine" workflow for co-developing multimodal data and models, validated on image-to-text, text-to-video (VBench top), and CLIP image-text pretraining.

Problem

Multimodal model development is split between model-centric and data-centric paths with no cheap, systematic infrastructure for jointly iterating on data processing and models, making experimentation costly and insights hard to attribute.

Approach

A sandbox that builds small single-operator and multi-operator data pools from Data-Juicer filters, trains reference models on them at controlled cost, uses model benchmark feedback to rank OPs and recipes, and transfers the resulting recipes to larger-scale training via a pyramid of data pools.

Key results

• On the VBench leaderboard (Sep 23, 2024), the Data-Juicer DJ 228k model reaches 82.53 Board Avg., ranking first above Gen-3 (82.32) and VEnhancer (81.97).
• MGM-2B trained on 159k instances (x4 repetition of a filtered pool) reaches +2.12% average performance over the baseline trained on 1226k instances.
• Top-ranked single-OP pools produce large gains on CLIP pre-training, e.g., +39.53% avg. change for the CLIP image-text similarity filter's high pool.

Measured effects reported in the paper

Surprising findings

• Combining top-ranked single operators does not always yield better recipes; adding high-performing OPs can reduce gains.
• Using the image-text similarity filter alone outperformed its combinations for CLIP pre-training, possibly acting as implicit distillation.

Limitations

• Results rely on averaged benchmark scores with small repetition counts (2 to 5), and many findings are task-specific.
• Theoretical cost analysis (Hoeffding bound) is loose and the cost ratio r is not measured precisely.
• Top VBench result uses proprietary-free but additional distillation on a base teacher model, so comparison fairness is limited.

Experimental setup

Datasets: Data-Juicer operator-filtered pools MGM pretraining dataset (~1.2M) DataComp candidate pools VBench prompts

Benchmarks: VBench TextVQA MMBench MME DataComp 40-task suite

Metrics: VBench Board Average VBench Uniform Average average relative performance change over baseline

Baselines: Gen-3 VEnhancer Kling CogVideoX-5B T2V-Turbo MGM-2B baseline on full data

Builds on: Data-Juicer DataComp VBench T2V-Turbo EasyAnimate

Closest papers by meaning

• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
• M*: A Modular, Extensible, Serving System for Multimodal Models (NeurIPS 2026 · similarity 0.87)
• Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models (NeurIPS 2025 · similarity 0.87)
• LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.87)
• SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.87)
• MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens (NeurIPS 2024 · similarity 0.87)