Research map / Architectures and efficiency / Quantization
BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents BlueLM-V-3B, a 3B multimodal LLM co-designed with mobile NPU deployment, using a relaxed aspect-ratio dynamic resolution scheme, token downsampling, batched/pipelined ViT encoding, chunked prefill, and mixed INT4/INT8 quantization.
Problem
Deploying multimodal LLMs on mobile phones is hard due to memory limits, slow NPU processing of long inputs, and the many image tokens produced by dynamic-resolution schemes.
Approach
A relaxed aspect-ratio matching rule that avoids exaggerated image upscaling in dynamic resolution (LLaVA-NeXT/InternVL 1.5 style), combined with hardware-aware system optimizations (batched patch encoding, CPU/NPU pipelining, chunked token prefill, mixed-precision quantization, decoupled ViT/LLM loading) for on-device inference.
Key results
- BlueLM-V-3B reaches an average of 66.1 on OpenCompass among models up to 10B, ranking second overall and beating MiniCPM-V-2.6 (65.2) and InternVL2-8B (64.1).
- On MediaTek Dimensity 9300 with 4-bit LLM weights, BlueLM-V-3B achieves 24.4 token/s generation, versus 4.9 token/s for MiniCPM-V 2.5 (8B, CPU).
- Relaxed aspect ratio matching improves BlueLM-3B's TextVQA from 52.7 (InternVL 1.5 scheme) to 56.2 and DocVQA from 28.7 to 31.3 under the same reduced-token setting; MTVQA reaches 32.7 vs. 20.7 for Qwen2-VL-2B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace InternVL 1.5 or LLaVA-NeXT resolution selection with relaxed aspect ratio matching (ablation) | BlueLM-3B with SigLIP-400M · VQAv2 val, TextVQA val, DocVQA val, OCRBench, ChartQA test | score: VQAv2: 78.3/77.7 -> 79.5; TextVQA: 52.7/51.4 -> 56.2; DocVQA: 28.7/29.6 -> 31.3; OCRBench: 338/351 -> 360; ChartQA: 16.8 | Helped |
| use relaxed aspect ratio matching instead of InternVL 1.5 or LLaVA-NeXT resolution selection (ablation) | MiniCPM-2B with SigLIP-400M · VQAv2 val, TextVQA val, DocVQA val, OCRBench, ChartQA test | score: VQAv2: 70.5/70.1 -> 71.8; TextVQA: 46.9/44.2 -> 49.4; DocVQA: 26.2/24.3 -> 27.3; OCRBench: 327/324 -> 343; ChartQA: 15.7 | Helped |
| process image patches in batches of 4 rather than 1, 2, or 6 (ablation) | BlueLM-V-3B on MediaTek Dimensity 9300 · 2:4 aspect ratio image encoding | image encoding latency | Helped |
| process input tokens in chunks of 128 rather than 32, 512, or 2048 (ablation) | BlueLM-V-3B on MediaTek Dimensity 9300 · 2048 input tokens; KV cache length 2048 | latency and output speed | Helped |
| use relaxed aspect ratio matching rather than LLaVA-NeXT resolution selection (ablation) | dynamic resolution processing · LLaVA 665k training images | selected aspect ratio: 29k cases | Mixed |
| use relaxed aspect ratio matching rather than InternVL 1.5 resolution selection (ablation) | dynamic resolution processing · LLaVA 665k training images | selected aspect ratio: 523k cases; 25k cases | Mixed |
| batch image patches before ViT processing (ablation) | BlueLM-V-3B training · image encoding | speedup: 10% | Helped |
| use batched image encoding with pipeline parallelism (ablation) | BlueLM-V-3B on MediaTek Dimensity 9300 · 2:4 aspect ratio image encoding | latency: 1.9 + 0.16 = 2.06s; 200ms | Helped |
| quantize LLM weights to INT4 while using INT8 ViT and projector weights | BlueLM-V-3B · MediaTek Dimensity 9300 deployment | generation throughput: 24.4 token/s | Helped |
| deploy BlueLM-V-3B on the NPU rather than MiniCPM-V 2.5 on CPU | mobile MLLM deployment · MediaTek Dimensity 9300 | throughput: 4.9 token/s -> 24.4 token/s | Helped |
Surprising findings
- Simply enlarging images does not improve accuracy; the relaxed aspect-ratio scheme reduced image tokens yet improved benchmark scores over LLaVA-NeXT and InternVL 1.5 schemes.
Limitations
- Mobile deployment evaluated on a single device (vivo X100 with Dimensity 9300) only
- Training data is largely in-house and not released, limiting reproducibility
- Comparison with MiniCPM-V uses numbers from the original paper under unspecified test configurations
Experimental setup
Datasets: LLaVA 558k ShareGPT4V 1200k ALLaVA 708k LLaVA 665k in-house image-text data PDF-derived image-text pairs GPT-4o and Gemini Pro generated captions/QA
Benchmarks: OpenCompass (MMBench, MMStar, MMMU, MathVista, HallusionBench, AI2D, OCRBench, MM-Vet) TextVQA DocVQA MTVQA VQAv2 ChartQA OCRBench
Metrics: accuracy average score latency (s) throughput (token/s) memory (GB)
Baselines: MiniCPM-V-2.6 InternVL2-8B Qwen2-VL-2B MiniCPM-V 2.5 Phi-3-Vision InternVL2-4B MiniCPM-V-2 POINTS-Qwen2.5 LLaVA-NeXT InternVL 1.5
Builds on: LLaVA-NeXT InternVL 1.5 VILA LLaVA
Closest papers by meaning
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.89)
- PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS (ICLR 2025 · similarity 0.87)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe (CVPR 2026 · similarity 0.87)
- A Comprehensive Overhaul of Multimodal Assistant with Small Language Models (AAAI 2025 · similarity 0.87)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.