Research map / Vision-language and multimodal / Human and embodied modeling

M*: A Modular, Extensible, Serving System for Multimodal Models

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents M*, a serving runtime that represents composite multimodal models as dataflow graphs (Walk Graph) with sequential, parallel, loop, and streaming primitives, and reports latency/throughput gains over vLLM-Omni and SGLang-Omni on BAGEL, Qwen3-Omni, Orpheus, and V-JEPA 2.

Problem

Existing LLM and multimodal serving systems assume fixed, mostly autoregressive execution structures and stage DAGs, so they cannot express non-AR loops, internal parallelism, request-specific component paths, or fine-grained placement for composite multimodal models.

Approach

Model composite architectures as a computation graph of components plus a finite set of named Walks per request, using Sequential, Parallel, Loop, DynamicLoop, and StreamingGraphEdge primitives, and compile this graph to a distributed runtime with per-component engines and flexible GPU placement.

Key results

• On BAGEL text-to-image, M* achieves about 20% lower average p50 E2E latency than vLLM-Omni; up to 2.64x lower on image editing with CFG parallelism.
• On Qwen3-Omni text-to-speech, M* reaches up to 2.9x lower RTF and 2.7x higher throughput than vLLM-Omni at batch size 16 (4.0x vs SGLang-Omni).
• On V-JEPA 2-AC rollouts, M* delivers p50 speedups of 2.08x, 3.76x, and 12.5x at horizons 4, 15, and 30 versus the native implementation.

Surprising findings

• No single vLLM-Omni configuration performs well across T2I, I2I, and I2T; the default config is poor on I2I and the single-stage config is poor on I2T.

Limitations

• Evaluated on only five models and a limited set of hardware configurations (single 4xH100 and 8xH200 nodes).
• Several baseline settings were excluded due to correctness or performance issues in baselines, limiting comparison.
• Does not yet explore sequence, context, or pipeline parallelism, or sparse attention optimizations.

Experimental setup

Datasets: VBench Food101 Seed-TTS DROID

Benchmarks: VBench Food101 Seed-TTS DROID

Metrics: p50 end-to-end latency p95 latency real-time factor throughput TTFT

Baselines: vLLM-Omni SGLang-Omni VoxServe Meta vjepa2 native implementation

Builds on: vLLM SGLang FlashInfer Mooncake continuous batching

Closest papers by meaning

• ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.88)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.88)
• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.87)
• Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.87)
• Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development (ICML 2025 · similarity 0.87)
• D-VLA: A High-Concurrency Distributed Asynchronous RL Framework for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)