LLM Inference Energy Efficiency Benchmark Dataset

A comprehensive benchmark dataset capturing power consumption, performance metrics, and timing traces for Large Language Models (LLMs) running on TensorRT-LLM inference engine using NVIDIA H100 GPUs. The dataset was produces for the paper SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference (ICPE 2026), consult it for any additional information.

If you do use this dataset in your research, please make sure to cite our paper.The dataset provides a notebook (dataset_usage.ipynb) as a usage example. The dataset is also available in our GitLab repository.
Overview

This dataset contains detailed measurements of 13 different LLM models across various workload configurations, providing insights into token generation performance, energy consumption, and throughput characteristics. Each model was benchmarked under multiple input/output token lengths combinations to enable thorough analysis of inference behavior.

Models Included

The dataset covers 13 open-source LLM models ranging from 1.3B to 9B parameters:

- Falcon: falcon-rw-7b
- Gemma: gemma-2-2b-it, gemma-2-9b-it
- Granite: granite-3b-code-base-2k, granite-8b-code-base-4k
- LLaMA: Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct
- OPT: opt-1.3b, opt-2.7b, opt-6.7b
- Qwen: Qwen2-1.5B, Qwen2-7B

Metrics

Each model's CSV file contains the following columns:

- Workload Configuration: Input Length, Output Length, Max Batch Size, Number of Requests
- Performance: Request Throughput, Total/Per User Output Throughput, Total Token Throughput
- Latency: Total Latency, Average Request Latency
- Energy: Energy (KJ), Energy Per Token (J), Efficiency (Tokens/J)
- System: Max Runtime Tokens

Trace Files (traces/*.csv)

Time-series data captured at 100ms~500ms intervals:

- Timestamp (ms): Time since benchmark start
- gpu_power: GPU Power Consumption (Watts)
- gpu_clock: GPU Clock Frequency (MHz)

The metrics were sampled using the NVIDIA Management Library Python API.

Citation
@inproceedings{Pizzini_Cavagna_2026,   title={SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference},   url={http://dx.doi.org/10.1145/3777884.3797011},   DOI={10.1145/3777884.3797011},   booktitle={Proceedings of the 17th ACM/SPEC International Conference on Performance Engineering},   publisher={ACM},   author={Pizzini Cavagna, Hiari and Proia, Andrea and Madella, Giacomo and Esposito, Giovanni Battista and Antici, Francesco and Cesarini, Daniele and Kiziltan, Zeynep and Bartolini, Andrea},   year={2026},   month=May, pages={83–95} }