Research map / Architectures and efficiency / RNNs and normalization
TINY BUT MIGHTY: A SOFTWARE-HARDWARE CO- DESIGN APPROACH FOR EFFICIENT MULTIMODAL IN- FERENCE ON BATTERY-POWERED SMALL DEVICES
Yilong Li, Yijing Zeng, Shuai Zhang, Hao Zhang, Xinmiao Xiong, Jingyu Liu, Pan Hu, Suman Banerjee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decompose LMM and offload vision encoder to NPU, LLM to GPU with zero-copy TABM | Nanomind · edge devices, end-to-end energy | energy: 42.3% reduction | Improved |
| cross-accelerator dynamic offloading vs official rkllm on Orange Pi 5 Ultra | Nanomind · Qwen2-VL-2B-Instruct InfoVQA | end-to-end latency: 36.2% reduction | Improved |
| zero-copy TABM ring buffer instead of llama.cpp copy-based layer offloading (ablation) | Nanomind · embedding transfer | memory usage and CPU load | Improved |
| run vision encoder on NPU instead of GPU or CPU (ablation) | Nanomind · SigLip and ArcFace latency | per-image latency | Improved |
| fused dequant-GEMM OpenCL kernel for LLM decoding (ablation) | Nanomind · Qwen2-1.5B-W8A8 text-only | throughput (tok/s) | Improved |
| offload more layers to GPU in llama.cpp (ablation) | llama.cpp · Llama-3-8B (2-bit) | memory usage: 2.9GB -> 5.5GB | Hurt |
| lower ViT precision in hybrid quantization (ablation) | Nanomind · MMBench, MME, InfoVQA | accuracy | Hurt |
| kernelized linear attention instead of quadratic attention on GPU (ablation) | Nanomind · tested benchmarks | accuracy | No clear effect |
| switch to on-demand cascade inference in critical battery state | Nanomind · 2000 mAh battery | runtime: up to 18.8 hours | Improved |
Closest papers by meaning
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.83)
- AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints (AAAI 2025 · similarity 0.83)
- HCInfer: An Efficient Inference System via Heterogeneous Error Compensation for Resource-Constrained Devices (NeurIPS 2026 · similarity 0.82)
- Analog Foundation Models (NeurIPS 2025 · similarity 0.82)
- ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism (NeurIPS 2025 · similarity 0.82)
- Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
