Replication Package for 'Data-Driven Analysis and Optimization of Machine Learning Systems Using MLPerf Benchmark Data'

Data-Driven Analysis and Optimization of Machine Learning Systems Using MLPerf Benchmark Data This repository contains the full replication package for the Master's thesis 'Data-Driven Analysis and Optimization of Machine Learning Systems Using MLPerf Benchmark Data'. The project focuses on leveraging public MLPerf benchmark data to analyze ML system performance and develop a multi-objective optimization framework for recommending optimal hardware configurations. The framework considers the trade-offs between three key objectives: 1. Performance (maximizing throughput) 2. Energy Efficiency (minimizing estimated energy per unit) 3. Cost (minimizing estimated hardware cost) Repository Structure This repository is organized as follows: Data_Analysis.ipynb: A Jupyter Notebook containing the code for the Exploratory Data Analysis (EDA) presented in the thesis. Running this notebook reproduces the plots in the eda_plots/ directory. Dataset_Extension.ipynb : A Jupyter Notebook used for the data enrichment process. It takes the raw `Inference_data.csv` and produces the Inference_data_Extended.csv by adding detailed hardware specifications, cost estimates, and derived energy metrics. Optimization_Model.ipynb: The main Jupyter Notebook for the core contribution of this thesis. It contains the code to perform the 5-fold cross-validation, train the final predictive models, generate the Pareto-optimal recommendations, and create the final result figures. Inference_data.csv: The raw, unprocessed data collected from the official MLPerf Inference v4.0 results. Inference_data_Extended.csv: The final, enriched dataset used for all analysis and modeling. This is the output of the Dataset_Extension.ipynb notebook. eda_log.txt: A text log file containing summary statistics generated during the exploratory data analysis. requirements.txt: A list of all necessary Python libraries and their versions required to run the code in this repository. eda_plots/: A directory containing all plots (correlation matrices, scatter plots, box plots) generated by the EDA notebook. optimization_models_final/: A directory where the trained and saved final model files (.joblib) are stored after running the optimization notebook. pareto_validation_plot_fold_0.png: The validation plot comparing the true vs. predicted Pareto fronts, as presented in the thesis. shap_waterfall_final_model.png: The SHAP plot used for the model interpretability analysis, as presented in the thesis. Requirements and Installation To reproduce the results, it is recommended to use a Python virtual environment to avoid conflicts with other projects. 1. Clone the repository: bash git clone <your-repository-url> cd <repository-directory> 2. **Create and activate a virtual environment (optional but recommended): bash python -m venv venv source venv/bin/activate # On Windows, use `venv\Scripts\activate` 3. Install the required packages: All dependencies are listed in the `requirements.txt` file. Install them using pip: bash pip install -r requirements.txt Step-by-Step Reproduction Workflow The notebooks are designed to be run in a logical sequence. Step 1: Data Enrichment (Optional) The final enriched dataset (`Inference_data_Extended.csv`) is already provided. However, if you wish to reproduce the enrichment process from scratch, you can run the **`Dataset_Extension.ipynb`** notebook. It will take `Inference_data.csv` as input and generate the extended version. Step 2: Exploratory Data Analysis (Optional) All plots from the EDA are pre-generated and available in the `eda_plots/` directory. To regenerate them, run the **`Data_Analysis.ipynb`** notebook. This will overwrite the existing plots and the `eda_log.txt` file. Step 3: Main Model Training, Validation, and Recommendation This is the core of the thesis. Running the Optimization_Model.ipynb notebook will execute the entire pipeline described in the paper: It will perform the 5-fold group-aware cross-validation to validate the performance of the predictive models. It will train the final production models on the entire dataset and save them to the optimization_models_final/ directory. It will generate the final Pareto front recommendations and single-best recommendations for the Computer Vision task. It will generate the final figures used in the results section, including pareto_validation_plot_fold_0.png and shap_waterfall_final_model.png.