Research map / Architectures and efficiency / Quantization
PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PalmBench, an automated USB-driven benchmark that measures efficiency, power, accuracy degradation, hallucination and toxicity of quantized LLMs (via MLC-LLM and llama.cpp) across Android, iOS, and Linux edge devices.
Problem
Local deployment of LLMs on mobile and edge devices requires balancing quality, latency, throughput, memory and power under hardware limits, but existing benchmarks target full-scale models on cloud GPUs and largely ignore quantization-induced hallucination and toxicity on mobile.
Approach
An lightweight, all-in-one automated benchmarking framework that collects resource, power and thermal metrics from phones and edge boards over USB, and evaluates quantized LLMs against their non-quantized outputs for accuracy, hallucination and toxicity.
Key results
- iPhones (e.g., iPhone 12 Pro) achieved higher throughput and lower resource utilization than Android devices and Jetson Orin Nano, per the paper's throughput analysis.
- For Llama3-8B, 2-bit quantization showed 34.7% hallucination versus 7.9% at 8-bit and 9.1% at 4-bit (GPTQ), with toxicity rising from 23.965 (8-bit) to 46.243 (2-bit).
- The 4-bit Llama-3.2-3B model consumed 25.2% more power than its 3-bit counterpart in the reported temperature/power study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use lower-bit quantization instead of higher-bit quantization (ablation) | Llama3-8B · Llama3-8B hallucination evaluation | Hallucination: 7.9% -> 34.7% | Hurt |
| Use 4-bit ggml rather than 4-bit GPTQ (ablation) | Llama3-8B · Llama3-8B hallucination evaluation | Hallucination: 9.1% -> 12.5% | Hurt |
| Use 4-bit quantization rather than 3-bit quantization (ablation) | Llama-3.2-3B · Mobile-device inference | Power Consumed: 13.32 -> 14.23; 12.98 -> 13.51; 14.54 -> 14.68; 13.25 -> 15.26; 11.21 -> 13.12; 10.13 -> 13.05; 25.4 -> 27.8; 22.3 -> 25 | Hurt |
| Use 4-bit rather than 3-bit quantization (ablation) | Llama-3.2-3B · Mobile-device inference | Peak Temp.: 47.8 -> 53.1; 53.2 -> 54.8; 52.1 -> 52.6; 52.8 -> 48.7; 47.3 -> 47.2; 45.3 -> 46.3; 71.5 -> 75.4; 61.5 -> 69.5 | Mixed |
| Use 3-bit rather than 4-bit quantization (ablation) | Mobile LLMs · Mobile inference | Memory footprint | Helped |
| Use 4-bit rather than 3-bit quantization (ablation) | Llama-3-8B-Instruct · Pixel 7 inference | GPU workload | Hurt |
| Use higher-bit quantization rather than lower-bit quantization (ablation) | Quantized LLMs · MTBench | MTBench score | Helped |
| Use 5-bit rather than 4-bit quantization (ablation) | Quantized LLMs · Output matching | Exact Match and F1 | Hurt |
| Use 4-bit rather than 3-bit quantization (ablation) | Llama-3.2-3B · Mobile-device inference | Power consumption: 25.2% more power | Hurt |
Surprising findings
- The 5-bit model showed greater performance degradation than all 4-bit variants, attributed to quantization algorithm and framework limitations.
- Higher-bit quantization sometimes consumed more power and raised temperature, despite memory reductions from quantization.
Limitations
- Mobile memory limits excluded larger models and higher-bit quantizations from many experiments.
- Hallucination and toxicity are judged by LLM judges (GPT-4o, Claude-3.5-Sonnet) rather than human annotation, and the paper gives limited detail on the judging protocol.
- The paper states code will be released but gives no link; several reported results are internally inconsistent (e.g., model/device lists differ across sections).
Experimental setup
Datasets: SQuAD Natural Questions MT-Bench HaluEval TruthfulQA Perspective API toxicity
Benchmarks: SQuAD Natural Questions MT-Bench HaluEval TruthfulQA
Metrics: exact match F1 score throughput (tok/s) memory footprint (GB) CPU/GPU utilization (%) power consumed (mWh) peak and average temperature hallucination rate (%) toxicity score
Baselines: MELT MobileAIBench Starting-kit competition framework
Builds on: MLC-LLM llama.cpp GPTQ AWQ
Closest papers by meaning
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.89)
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.89)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.89)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.89)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.88)
- Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.