Einsum Benchmark: Enabling the Development of Next-Generation Tensor Execution Engines

Modern artificial intelligence and machine learning workflows rely on efficient tensor libraries. However, tuning tensor libraries without considering the actual problems they are meant to execute can lead to a mismatch between expected performance and the actual performance. Einsum libraries are tuned to efficiently execute tensor expressions with only a few, relatively large, dense, floating-point tensors. But, practical applications of einsum cover a much broader range of tensor expressions than those that can currently be executed efficiently. For this reason, we have created a benchmark dataset that encompasses this broad range of tensor expressions, allowing future implementations of einsum to build upon and be evaluated against. In addition, we also provide generators for einsum expression and converters to einsum expressions in our repository, so that additional data can be generated as needed. The benchmark dataset, the generators and converters are released openly and are publicly available at https://benchmark.einsum.org.

The broader data collection process included contributions from individuals whose data was transformed. We duly acknowledge the following for making their data publicly available:

Fichte, Johannes; Hecher, Markus; Florim Hamiti: Model Counting Competition 2020

Fichte, Johannes; Hecher, Markus: Model Counting Competition 2021 2022 2023

Fichte, Johannes; Hecher, Markus; Woltran, Stefan; Zisser, Markus: A Benchmark Collection of #SAT Instances and Tree Decompositions

Meel, Kuldeep S.: Model Counting and Uniform Sampling Instances

Automated Reasoning Group at the University of California, Irvine: UAI Competitions

Martinis, John M. et al.: Quantum supremacy using a programmable superconducting processor Dataset. Dryad.

Moreover, we thank the following authors of open source software used to generated instances:

Gray, Johnnie: quimb, cotengra

Soos, Mate, Meel, Kuldeep S: Arjun

Stoian, Mihail: Netzwerk

Liu, Jinguo; Lua, Xiuzhe; Wang, Lei: Yao.jl

Liu, Jinguo: YaoToEinsum.jl