An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode

<strong>Abstract</strong><br> AlphaCode, the code generation system by DeepMind, is an AI technology for assisting software developers in solving competitive programming problems using natural language problem descriptions.<br> However, there is no existing work comparing AlphaCode-generated codes with human codes.<br> In this paper, we conduct an empirical study to find code similarities and performance differences between AlphaCode-generated codes and human codes.<br> We collect 44 generated codes in C++ and Python languages from AlphaCode official website that solve 22 problems on Codeforces.<br> We then retrieve 31,736 human codes by using Codeforces API.<br> The results show that (i) the generated code from AlphaCode and the human code is not similar (i.e., the similarity of 0.26 for both C++ and Python), however, the code fragments in the generated code are comprised of various human codes (i.e., uniqueness of 3.30\% and 8.94\% for C++ and Python respectively) and<br> (ii) the generated code performs on par with or worse than the human code in terms of execution time and memory usage.<br> Moreover, AlphaCode employs excessive nested loops and unnecessary variable declarations (e.g., used \textit{long long} instead of \textit{int}), which causes a low performance regarding our manual investigation.