Research map / Vision-language and multimodal / Human and embodied modeling
Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Data-Juicer 2.0 is an open-source, multimodal data processing system with 100+ operators, multi-level interfaces (Python, RESTful, web, natural language) and an adaptive runtime that scales from standalone to 10k+ CPU-core cloud clusters.
Problem
Foundation-model data processing needs multimodal (text, image, video, audio) operators and scalable, usable systems, but existing frameworks are text-only, big-data engines are ill-suited to per-sample model-heavy operations, and ecosystems are fragmented across APIs.
Approach
A layered system with a unified Data-Juicer-Dataset facade over Hugging Face Datasets, Ray Data and MaxFrame, decoupled operators with adaptive batching, parallelism and OP reordering, operator-level checkpointing, and multiple user interfaces including ReAct-based natural language agents.
Key results
- Ray-DLC with 3200 cores processes 500x and 2500x datasets in 1780.86s and 7083.5s; MaxCompute needs about 1.5x more time for multimodal recipes.
- Subset splitting for Ray reduces processing time of the 12,500x dataset from over 5,000s to about 2,000s with 100 nodes and 12,800 cores (2x-3x speedup).
- RayDeduplicator processes 5TB in 2.8 hours on 8 Ray nodes (8*160 cores), versus NeMo Curator's 1.8 hours on 1.1TB with 64 A100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MinHash-based RayDeduplicator with load-balanced union-find and hash-based aggregation instead of vanilla Ray | Data-Juicer 2.0 deduplication · Deduplication | speedup: 3.3x speedup | Helped |
| Increase RayDeduplicator resources from 4*160 to 8*160 CPU cores | RayDeduplicator · Redpajama/Common Crawl, 200GB–5TB | deduplication time: 58.9%$sim$67.1% of the original time | Helped |
| Use AI-oriented CPFS instead of standard CPFS | Ray processing · 12,500x dataset, 3200 cores | processing time: 4,396s -> 1,625s; 2.7x faster | Helped |
| Use NAS or OSS instead of standard CPFS | Distributed data processing · Large-scale datasets | processing time: 20-30% | Hurt |
| Enable adaptive subset splitting for Ray | Ray mode · Large datasets | processing time: 2x$\sim$3x acceleration | Helped |
| Enable adaptive subset splitting instead of Ray's automatic block splitting | Ray mode · 12,500x dataset, 100 Ray-DLC nodes, 12,800 cores | processing time: over 5,000s to about 2,000s | Helped |
| Pre-split datasets according to size and node count | Ray mode · Large-scale distributed processing | peak network I/O: about 160MB/s to 60MB/s | Helped |
| Use Ray-DLC rather than MaxCompute for multimodal recipes | Data-Juicer 2.0 · 70B-sample-scale multimodal recipes | processing time: MaxCompute requires 1.5 times more processing time | Helped |
| Use MaxCompute rather than Ray for text-only recipes | Data-Juicer 2.0 · 70B-sample-scale text-only recipes | processing time: about 1/4 of the time and using 1/2 fewer cores | Helped |
| Use Ray on DLC rather than native ECS nodes | Ray mode · Medium-scale datasets, 56M samples | processing time: 24.8% | Helped |
| Increase Ray node count for small-scale multimodal processing to 4 nodes | Ray mode · Small-scale multimodal datasets | speedup: 138% and 226% with 4 nodes | Helped |
| Increase Ray nodes for small-scale text-only processing | Ray mode · Small-scale text-only datasets | processing time: 148% | Mixed |
Surprising findings
- For small-scale text data, adding Ray nodes yields smaller speedups or even increased processing time due to I/O and communication costs, so standalone mode is often sufficient.
- MaxCompute is about 4x faster than Ray for text-only recipes at 70B-sample scale, attributed to its co-optimization of compute and storage.
Limitations
- Future work needed to optimize single-node transmission bottleneck in Ray's head node.
- Multilingual support and additional domains (AI4science, embodied intelligence) are not yet covered.
- Non-linear scaling tradeoffs depend on data modality and access patterns, and storage-compute co-design matters beyond 10M samples.
Experimental setup
Datasets: RedPajama Common Crawl
Metrics: processing time speedup scalability
Baselines: NVIDIA NeMo Curator
Builds on: Data-Juicer 1.0 ReAct AgentScope Hugging Face Datasets Ray Data
Closest papers by meaning
- Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development (ICML 2025 · similarity 0.87)
- Griffin: Towards a Graph-Centric Relational Database Foundation Model (ICML 2025 · similarity 0.81)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.81)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.81)
- Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (ICLR 2024 · similarity 0.81)
- Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets (NeurIPS 2026 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.