NATS-Bench-MCU

NATS-Bench-MCU: Hardware Measurements for NATS-Bench TSS on Nordic nRF5340 This dataset accompanies the paper "NATS-Bench-MCU: A Tabular Hardware Benchmark for Neural Architecture Search on Microcontrollers" (AutoML 2026). It provides direct, physically measured hardware metrics for all 15,625 architectures in the NATS-Bench topology search space (TSS), deployed on a Nordic Semiconductor nRF5340 microcontroller. No values are simulated or estimated. For each deployable architecture, the dataset reports flash and SRAM footprint, inference latency, mean current draw, mean power, and energy per inference, measured via a Nordic Power Profiler Kit II at 100 kHz sampling rate. Of the 15,625 architectures, 14,279 (91.4%) were successfully deployed and profiled; 1,346 (8.6%) failed a flash-feasibility precheck because their INT8 flatbuffer exceeded the 800 KB threshold. Hardware-aware neural architecture search depends on benchmarks that expose not only predictive performance, but also the deployment costs of candidate architectures on the target device. Existing tabular NAS benchmarks have enabled reproducible algorithm development at low computational cost, and hardware-aware extensions have added latency and energy measurements for several edge-class platforms. However, microcontroller-class devices remain largely absent from these resources, despite being among the most constrained and practically relevant deployment regimes for Edge AI. We present NATS-Bench-MCU, a tabular hardware benchmark that augments the NATS-Bench topology search space with end-to-end measurements on a Nordic nRF5340 microcontroller. For each deployable architecture, we report flash and SRAM footprint, inference latency, current draw, and energy per inference, measured directly on physical hardware using a Nordic Power Profiler Kit II. In contrast to simulation- or proxy-based hardware costs, all reported on-device metrics are obtained from real deployments through a reproducible firmware, quantization, and measurement pipeline. Files export/ — Publication CSVs (4.7 MB total)The primary entry point for most users. Contains all data needed to reproduce the paper's figures and tables. No decompression or special tooling required — plain CSV files readable by any spreadsheet application or pandas. hw_metrics.csv (4.0 MB, 14,279 rows) — Core hardware measurements for every successfully deployed architecture: architecture index, cell topology string, INT8 model size, total ROM and RAM usage, inference latency, mean current, mean power, energy per inference, board ID, tensor arena usage, operator count, and Zephyr memory-report breakdowns by category. accuracies.csv (666 KB, 14,279 rows) — NATS-Bench test accuracies (CIFAR-10, CIFAR-100, ImageNet-16-120) for each successfully deployed architecture, sourced from the NATS-Bench API under the 200-epoch training protocol.failures.csv (51 KB, 1,346 rows) — One row per failed architecture: index, INT8 model size, and the size reported in the precheck error message.coverage.csv (157 B, 5 rows) — Pipeline funnel: how many architectures reached each stage (converted → submitted → successful / failed). README.md — Column-by-column documentation for all four CSVs, including units, measurement method, and a figure-to-column mapping. artifacts_XXXXX-YYYYY.tar.gz — Raw measurement archives (≈ 2.2 GB each, 35.1 GB total)16 chunked tar.gz archives containing the raw per-architecture output directories for all 15,625 architectures (1,000 architectures per archive, except the last which contains 625). Each architecture directory holds: results.json (aggregated metrics), ppk2_summary.csv (per-inference power summary), ppk2_samples.parquet (raw PPK2 current trace at 100 kHz), uart.log (TFLite Micro runtime log with operator count and tensor arena usage), rom.json.gz / rm.json.gz (Zephyr memory reports), flash.log.gz (firmware build log), model.cpp.gz (generated C model array), and the INT8 .tflite flatbuffer. Failed architectures contain only meta.json, error.log, and the .tflite file. You do not need to download all 16 archives. Use index.json and dataset_example.py (downloadable from our GitHub Repo) to access a single architecture by downloading only its archive (≈ 2.2 GB). index.json (588 KB)Maps every architecture index (0–15624) to the archive filename that contains it. Required by dataset_example.py for on-demand single-archive access. dataset_example.pyMinimal Python example script (Python 3.8+, stdlib only except optional pyarrow). Two commands: To access the files conveniently, please have a look at the our corresponding github repository: GitHub: NATS-Bench-MCU You will find a folder called "dataset" there, which contains exmaple scripts to access single evaluations from the dataset as well as in-depth explanations about the code Reproduce all paper figures (export CSVs only, no archives needed): git clone https://github.com/ankilab/NATS-Bench-MCUpip install matplotlib numpypython plot_from_export.py --data export/ --out figures/ Access a single architecture's raw data (No need to decompress any files): (Download the daraset_example.py by cloning the HW-NATS-Bench GitHub Repo and place all the files from Zenodo in the dataset folder) python dataset_example.py print 42python dataset_example.py export 42 --out ./evaluation_42/ License: CC BY 4.0