Multi-layer Sliced Design and Analysis with Application to AI Assurance

Enhancing AI assurance in tuning configurations and hyper-parameters of AI algorithms is an important problem in many applications. This work provides an experimental design method to address this challenging problem. The key idea of the method is to conduct an efficient experimental design to detect and quantify the effects of hyperparameters on the performance of AI algorithms. Specifically, the method proposes a multi-layer sliced design to enable quantifying the effects of slice factors and design factors to account for hyperparameters having different effects under different configurations of the AI algorithm. Moreover, this method develops an effective analysis procedure to estimate the effects of these factors and test their significance. The performance of the proposed design and analysis methods is successfully illustrated by simulation studies and real-world AI applications.