Environment-aware Dynamic Partitioning of Neural Networks - Research Data and Code

Supporting data for "Environment-aware Dynamic Partitioning of Neural Networks".

We present a method to dynamically partition neural networks on resource-constrained IoT devices to increase FPS and decrease total inference latency. Currently, in real-time neural network systems, input data collected from sensors is transmitted to local servers for inference. Network transmission often introduces significant latency. In feedforward neural networks, the output of each layer is passed to the next layer, with data size progressively decreasing through successive layers. Instead of transmitting the original data, we exploit this downward size trend by using an edge device (either co-located with the sensor or embedded within it) to transmit smaller intermediate results to the local server, which processes the remaining layers. Furthermore, we quantize and compress these intermediate results using various techniques to reduce their size before transmission by up to 97.11%. Even with lossy compression, we maintain a mean average precision (mAP) of 0.52.

Includes a partition point calculator to calculate the best partition point for the YOLOv8 and YOLOv3-tiny neural networks. Included data is for Nvidia Jetson Orin Nano 8GB (YOLOv8) and Nvidia Jetson Nano 2GB (YOLOv3-tiny). Code for measuring the data is also included.