Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility

Interactive computational notebooks (e.g., Jupyter notebooks) are widely used in machine learning engineering (MLE) to program and share end-to-end pipelines, from data preparation to model training and evaluation. However, environment erosion—the rapid evolution of hardware and software ecosystems for machine learning—has rendered many published MLE notebooks non-reproducible in contemporary environments, hindering code reuse and scientific progress. To quantify this gap, we study 12,106 notebooks mined from 75 popular Kaggle competitions: only 26% remain reproducible today. Crucially, we find that environment backporting, i.e., downgrading dependencies to match the submission time, does not improve reproducibility but rather introduces additional failure modes. To address environment erosion, we design and implement MLEModernizer, an LLM-driven agentic framework that treats the contemporary environment as a fixed constraint and modernizes notebook code to restore reproducibility. MLEModernizer iteratively executes notebooks, collects execution feedback, and applies targeted fixes in three types: error-repair, runtime-reduction, and score-calibration. Evaluated on 8,210 notebooks that are non-reproducible under the baseline environment, MLEModernizer makes 3,413 (41.5%) reproducible. MLEModernizer enables practitioners to validate, reuse, and maintain MLE artifacts as the hardware and software ecosystems continue to evolve.====================================================================================================================== Offline Grading: https://github.com/openai/mle-bench. Git clone https://github.com/openai/mle-bench.git or unzip mle-bench-grader.tar.gz and then follow the instruction to install, e.g., pip install -e . and mlebench prepare --all, etc. The per-fix CSV outputs for each file-level and cell-level modernization (i.e., ./results/upgrade/{gpt_cell, gpt_file, oss_file}/csv_output/fix_{fix_number} for fix_number ∈ {1,…,16}) are not included in this replication package due to size constraints. However, when applicable, we have recorded the score for each fix---consistent with MLE-Bench requirement---in ./results/upgrade/{gpt_cell, gpt_file, oss_file}/executable_files_w_timer_parrallel.json , and we have included all reproducible CSVs at ./results/upgrade/{gpt_cell, gpt_file, oss_file}/csv_output/{notebookName}.csv where notebookName is made of {competitionName}_{userName}_{submissionName}_{versionNumber}_C1.ipynb. If the per-fix CSV outputs are required, we would recommand replicating the corresponding notebook(s) to generate the CSV again.