Research map / Architectures and efficiency / Neural architecture search
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HW-GPT-Bench, a surrogate benchmark for hardware-aware multi-objective NAS over GPT-2-style language models, predicting perplexity (weight-sharing supernets) and calibrated latency/energy (AutoGluon) across 13 devices.
Problem
Finding Pareto-optimal language model architectures under hardware metrics (latency, energy, memory, perplexity) is costly because exhaustive training and on-device measurement across many devices is infeasible; existing NAS benchmarks focus on CNNs/vision and ignore calibrated hardware noise for ca
Approach
Combines weight-sharing supernetworks (sandwich-rule training) for perplexity proxies with calibrated surrogate predictors (AutoGluon, mean and std models) for heteroscedastic latency and energy measurements across many GPUs and CPUs, enabling simulated multi-objective NAS runs in seconds.
Key results
- Perplexity and memory MLP surrogates reach rank correlation > 0.90 on 2000 unseen architectures.
- AutoGluon gives the best accuracy and calibration for H100 latency and RTX2080 energy (e.g., H100 latency MAE 0.153, R2 0.999; RMS calibration 0.223), outperforming LightGBM, XGBoost and mixed ensembles.
- Simulated multi-objective NAS runs on the benchmark complete in seconds on a single CPU; EHVI and NSGA-II reach higher hypervolume early than local and random search in the 2-objective setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoGluon instead of Ensemble (Mix) | latency surrogate · H100 | MAE: 0.569 -> 0.153 | Helped |
| Use AutoGluon instead of Ensemble (XGB) | latency surrogate · H100 | MAE: 0.620 -> 0.153 | Helped |
| Use AutoGluon instead of Ensemble (LGB) | latency surrogate · H100 | MAE: 0.361 -> 0.153 | Helped |
| Use AutoGluon instead of Ensemble (Mix) | energy surrogate · RTX 2080 | MAE: 1.830 -> 1.80 | Helped |
| Use AutoGluon instead of Ensemble (XGB) | energy surrogate · RTX 2080 | MAE: 1.832 -> 1.80 | Helped |
| Use AutoGluon instead of Ensemble (LGB) | energy surrogate · RTX 2080 | MAE: 2.094 -> 1.80 | Helped |
| Use AutoGluon instead of Ensemble (Mix) | latency surrogate · H100 | RMS Cal.: 0.472 -> 0.223 | Helped |
| Use AutoGluon instead of Ensemble (XGB) | latency surrogate · H100 | RMS Cal.: 0.481 -> 0.223 | Helped |
| Use AutoGluon instead of Ensemble (LGB) | latency surrogate · H100 | RMS Cal.: 0.559 -> 0.223 | Helped |
| Use AutoGluon instead of Ensemble (Mix) | energy surrogate · RTX 2080 | RMS Cal.: 0.298 -> 0.244 | Helped |
| Use AutoGluon instead of Ensemble (XGB) | energy surrogate · RTX 2080 | RMS Cal.: 0.286 -> 0.244 | Helped |
| Use AutoGluon instead of Ensemble (LGB) | energy surrogate · RTX 2080 | RMS Cal.: 0.347 -> 0.244 | Helped |
Surprising findings
- Energy and latency correlation with FLOPS weakens at larger model scales, mainly due to higher measurement noise.
- Bayesian optimization methods perform worse than random search and local search in the 3-objective setting.
Limitations
- Perplexity relies on inherited supernet weights rather than independently trained architectures.
- Hardware metrics are surrogate predictions with batch size and sequence length fixed, limiting generality.
- Large-scale energy and latency noise lowers correlations at bigger scales, affecting reliability of trade-off rankings.
Experimental setup
Datasets: OpenWebText hardware latency and energy measurements on 13 devices
Benchmarks: HW-GPT-Bench surrogate predictions (perplexity, latency, energy, memory)
Metrics: MAE RMSE MDAE MARPD R2 Spearman/Pearson correlation RMS calibration error MA calibration error miscalibration area hypervolume
Baselines: Random Search Local Search NSGA-II MOREA MOASHA EHVI LightGBM ensemble XGBoost ensemble
Builds on: GPT-2 Once-for-all AutoGluon sandwich rule Syne Tune
Closest papers by meaning
- PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS (ICLR 2025 · similarity 0.87)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.87)
- Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models (NeurIPS 2025 · similarity 0.87)
- BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models (NeurIPS 2024 · similarity 0.86)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.86)
- BitDelta: Your Fine-Tune May Only Be Worth One Bit (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.