Research map / Large language models / Agents, code and math
Einsum Benchmark: Enabling the Development of Next-Generation Tensor Execution Engines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a benchmark of 168 einsum problems across seven categories plus generators and converters, and uses it to expose performance pitfalls in current einsum libraries.
Problem
Current einsum libraries are tuned for a few large, dense, floating-point tensors, while practical einsum expressions (tensor networks, hyperedges, Hadamard products, traces, sparse and integer tensors) are much broader and lack a public benchmark.
Approach
A curated and generated benchmark of diverse real-world and synthetic einsum expressions with metadata and precomputed contraction paths, plus generators and converters, used to probe concrete weaknesses of opt_einsum, cotengra, NumPy, PyTorch, TensorFlow, JAX and SQLite.
Key results
- Minimizing intermediate size instead of operation count gave average execution time 1.78 s versus 6.37 s for minimizing operations on a quantum circuit problem across 100 paths each.
- For mc_2020_082, hybrid CPU-GPU execution reached 3.28 s versus 4.71 s CPU-only and 6.48 s GPU-only (1.44x speedup over CPU).
- For int16 problem mc_rw_blockmap_05_01.net, most backends fail; PyTorch float32 ran in 5.18 s while int64 took 41.77 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse tensor storage instead of dense tensors | einsum execution on PyTorch versus SQLite · mc_2021_036 | execution time (seconds): 43.2 ± 0.09 -> 34.2 ± 0.04 | Helped |
| Optimize contraction paths for intermediate size rather than operation count | opt_einsum with PyTorch backend · qc_circuit_n49_m14_s9_e6_pEFGH_simplified | average execution time (seconds): 6.37 s (SD: 2.68) -> 1.78 s (SD: 0.27) | Helped |
| Use GPU-only execution instead of CPU-only execution | PyTorch · mc_2020_082 | execution time (seconds): 4.71 ± 0.03 -> 6.48 ± 0.04 | Hurt |
| Use hybrid CPU-GPU execution for pairwise contractions | PyTorch · mc_2020_082 | execution time (seconds): 4.71 ± 0.03 -> 3.28 ± 0.01 | Helped |
| Use PyTorch instead of TensorFlow for floating-point pairwise contractions | opt_einsum backends · mc_rw_blockmap_05_01.net | execution time (seconds): 42.30 ± 0.11 -> 7.93 ± 0.29 | Helped |
| Use JAX int16 instead of JAX int32 | JAX · mc_rw_blockmap_05_01.net | execution time (seconds): 25.27 ± 0.13 -> 220.18 ± 0.91 | Hurt |
Surprising findings
- SQLite sparse execution (34.2 s) outperformed PyTorch dense execution (43.2 s) on a sparse model counting problem.
- Minimizing number of operations often yields slower execution than minimizing intermediate size due to skewed contractions.
Limitations
- Excludes problems too large for a single compute node and trivial small expressions.
- Limited to portable primitive data types.
- Hybrid execution uses a simple threshold policy and a custom loop rather than a library implementation.
Experimental setup
Datasets: UAI 2022 Competition graphical model instances model counting competition instances (2020-2023) weighted model counting competition instances opt_einsum random equations cotengra generators
Benchmarks: Einsum Benchmark
Metrics: execution time in seconds path computation time theoretical number of operations speedup
Baselines: opt_einsum cotengra NumPy einsum PyTorch TensorFlow JAX SQLite
Builds on: opt_einsum cotengra
Closest papers by meaning
- Exploiting Dynamic Sparsity in Einsum (NeurIPS 2025 · similarity 0.91)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.86)
- The Price of Freedom: Exploring Expressivity and Runtime Tradeoffs in Equivariant Tensor Products (ICML 2025 · similarity 0.84)
- Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.84)
- Activation Map Compression through Tensor Decomposition for Deep Learning (NeurIPS 2024 · similarity 0.84)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.