Research map / Architectures and efficiency / RNNs and normalization
The Hidden Joules: Evaluating the Energy Consumption of Vision Backbones for Progress Towards More Efficient Model Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures inference energy consumption and accuracy of about 1,200 timm ImageNet classification models on A100/H100 GPUs under PyTorch FP32 and TensorRT FP16, and releases an energy-efficiency scoring system and interactive web app.
Problem
Inference energy consumption of vision classification models is poorly understood; prior studies covered few models, used batch size 1, relied on outdated GPUs, or estimated energy from TDP and FLOPs, which is inaccurate.
Approach
Large-scale, batch-size-saturated, nvidia-smi-based energy measurement across ~1,200 timm models on two GPUs and two inference stacks, plus a weighted accuracy-energy score and throughput-based energy estimation (throughput implies the GPU is near TDP).
Key results
- Energy spans four orders of magnitude across models, with accuracy gains flattening: about 40% to 80% for the first 10x energy increase, then only +7 and +3 points for subsequent tenfold increases.
- Naive FLOPs-over-TDP energy estimation underestimates measured energy by a geometric mean factor of about 3.13 on A100, and energy can be estimated from throughput alone with 0.97% MAPE.
- TensorRT FP16 reduces energy by a geometric mean factor of 3.89 on A100 (4.02 on H100) relative to PyTorch FP32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TensorRT FP16 instead of PyTorch FP32 | ImageNet models on A100 · ImageNet | energy consumption: 3.89 | Helped |
| Use TensorRT FP16 instead of PyTorch FP32 | ImageNet models on H100 · ImageNet | energy consumption: 4.02 | Helped |
| Estimate energy using FLOPs divided by GPU FLOPs/second and multiplied by TDP | ImageNet models on A100 · ImageNet | energy estimation error: 3.13 and 3.16 | Hurt |
| Estimate energy using parameters, Gmacs, and activations | ImageNet models on A100 · ImageNet | MAPE: 34.73% | Helped |
| Infer energy consumption from throughput | ImageNet models on A100 · ImageNet | MAPE: 0.97% | Helped |
| Increase batch size to improve GPU utilization | EfficientViT on H100 · ImageNet inference | energy consumption per image | Helped |
| Increase batch size beyond full hardware utilization | EfficientViT on H100 · ImageNet inference | energy consumption per image | Mixed |
| Increase input image size (ablation) | Models supporting variable input sizes · ImageNet | energy consumption | Hurt |
| Increase input image size (ablation) | Models supporting variable input sizes · ImageNet | accuracy | No clear effect |
| Use RTX 4090 instead of RTX 3090 | 128 most efficient H100 models · ImageNet | geometric mean energy reduction: 21.0% | Helped |
| Use RTX 5090 instead of RTX 4090 | 128 most efficient H100 models · ImageNet | energy reduction: 5.7% | Helped |
Surprising findings
- Naive FLOPs/TDP energy estimates from prior work underestimate measured energy by roughly 3x and up to ~40x.
- RTX 5090 improved energy efficiency over RTX 4090 by only 5.7%, suggesting diminishing generational gains.
Limitations
- Results are measured only on NVIDIA GPUs (A100, H100 and consumer GPUs), with energy from nvidia-smi onboard sensors rather than external metering.
- Energy-accuracy trends are on ImageNet-style classification and may not transfer to downstream tasks such as detection or segmentation.
- The parameter/GMACs/activation regression is described by the authors as not highly accurate (34.73% MAPE).
Experimental setup
Datasets: ImageNet-1K validation ImageNet Real Labels ImageNet V2 Matched Frequency ImageNet Sketch ImageNet Adversarial ImageNet Rendition
Benchmarks: ImageNet-1K ImageNet Real Labels ImageNet V2 ImageNet Sketch ImageNet Adversarial ImageNet Rendition
Metrics: energy per image (J/img) top-1 accuracy throughput (img/s) mean absolute percentage error energy efficiency score Pearson correlation Spearman correlation
Baselines: Desislavov et al. FLOPs-TDP estimation Henderson et al. batch-size-1 measurements VGG
Builds on: Green AI Desislavov et al. energy estimation Henderson et al. energy reporting framework
Closest papers by meaning
- The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (NeurIPS 2025 · similarity 0.87)
- Energy-Based Transformers are Scalable Learners and Thinkers (ICLR 2026 · similarity 0.85)
- Position: Tensor Networks are a Valuable Asset for Green AI (ICML 2024 · similarity 0.85)
- E4: Energy-Efficient DNN Inference for Edge Video Analytics via Early Exiting and DVFS (AAAI 2025 · similarity 0.84)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.83)
- Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.