Research map / Large language models / AI and society
MeRino: Entropy-Driven Design for Generative Language Models on IoT Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeRino, sub-100M generative language models designed by maximizing a transformer information-entropy objective under compute constraints via an evolutionary search solved on CPU in minutes.
Problem
Scaling generative LLMs down for resource-constrained IoT and mobile devices needs heavy effort and domain knowledge, and existing NAS and zero-shot proxies are costly or poorly suited to language models.
Approach
Defines a network entropy from singular values of weight matrices, adds a depth-to-width effectiveness penalty for transformers, and maximizes this entropy under compute budgets with an evolutionary algorithm using a precomputed lookup table, so no GPU or training is needed during design.
Key results
- MeRino-64M matches OPT-350M average zero-shot accuracy (0.408 vs 0.408) with about 82% fewer parameters and 78% fewer FLOPs.
- Entropy design search takes about 0.05 h on Jetson Nano CPU versus 1.2 h on a GTX 1080Ti for TE-NAS, with an average accuracy of 0.408 vs 0.389.
- Entropy score achieves Spearman rho above 0.8 and Kendall tau above 0.6 with perplexity across 81 sampled architectures trained from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add block-wise parameter sharing (ablation) | MeRino · 60G FLOPs target | average accuracy: 0.381 -> 0.383 | Helped |
| add block-wise parameter sharing (ablation) | MeRino · 110G FLOPs target | average accuracy: 0.395 -> 0.392 | Hurt |
| add block-wise parameter sharing (ablation) | MeRino · 160G FLOPs target | average accuracy: 0.403 -> 0.408 | Helped |
| add effectiveness constraint γ (ablation) | MeRino · 110G FLOPs target | average accuracy: 0.360 -> 0.384 | Helped |
| add weighted entropy α (ablation) | MeRino with effectiveness constraint · 110G FLOPs target | average accuracy: 0.384 -> 0.392 | Helped |
| use INT8 quantization | MeRino · zero-shot learning tasks | accuracy | No clear effect |
| use INT4 quantization | MeRino · zero-shot learning tasks | accuracy: 1% | Hurt |
Surprising findings
- A deep 24-layer model with higher entropy obtained worse perplexity than a 1-layer wide model at similar parameter count, motivating the depth-width effectiveness constraint.
Limitations
- Entropy-based proxy and search constrained to sub-100M models; the paper itself reports MeRino-64M at 64M params while claiming sub-100M regime for mobile use.
- Comparisons with MobileLLM were not possible since its weights and evaluation were unreleased.
- Some headline claims (e.g., speedup 4.9x vs 4.6x average) are reported inconsistently across sections.
Experimental setup
Datasets: The Pile One Billion Word (LM1B) WikiText-2 Penn Treebank HellaSwag WinoGrande OpenBookQA ARC PubmedQA LogiQA BoolQ CB
Benchmarks: WikiText-2 Penn Treebank HellaSwag WinoGrande OpenBookQA ARC-E ARC-C PubmedQA LogiQA BoolQ CB WIC
Metrics: perplexity average accuracy latency FLOPs Spearman rank correlation Kendall tau
Baselines: OPT-125M OPT-350M Pythia-70M Pythia-162M Cerebras-111M GPT-2-124M TE-NAS DSS-Score LTS
Builds on: TE-NAS TF-TAS LTS OPT
Closest papers by meaning
- Rethinking Optimization and Architecture for Tiny Language Models (ICML 2024 · similarity 0.84)
- PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS (ICLR 2025 · similarity 0.84)
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.83)
- Language Modeling by Language Models (NeurIPS 2025 · similarity 0.83)
- Entropy-informed Decoding: Adaptive Information-Driven Branching (ICML 2026 · similarity 0.82)
- The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.