[Replication Package] Reasoning About Bugs in Learners' Scratch Programs Using Large Language Models

Replication package for our ICSE-SEET 2026 paper ‘Reasoning About Bugs in Learners’ Scratch Programs Using Large Language Models’https://doi.org/10.1145/3786580.3786949 Program analysis tools for Scratch support learners by identifying blocks indicative of bugs, which may be evidence of misconceptions and likely lead to faulty program behaviour. However, this information may not be sufficient to guide learners towards fixing both their misunderstandings and their programs. In this paper we therefore propose to use large language models (LLMs) as a means to explain the issues found by the program analysis, and to help learners understand how to debug and fix these issues. While the power of modern LLMs has resulted in a surge of new support tools for text-based programming languages, Scratch programs pose additional challenges for LLMs: The visual, block-based code first needs to be converted to textual prompts for the LLMs, and even then is it unclear whether the LLMs can exhibit similar performance as for text-based programming languages with their abundant training data. We determine the best way of generating prompts from Scratch programs and their bugs, and use these prompts with a dataset of buggy Scratch programs to evaluate the ability of LLMs to reason about these bugs. Our experiments demonstrate that LLMs can reason well about bugs in Scratch programs, thus setting the scene for more powerful tools that assist young learners in debugging their programs. ----- The contents of the dataset and instructions on how to run the scripts can be found in the README.md file.