On the Applicability of Language Models to Block-Based Programs

This dataset contains, on the one hand, software and data to reproduce the study, on the other hand, the raw data of the experiments of the evaluation as conducted in the paper. This file shall provide you with a general overview on the dataset: <strong>n-gram model</strong>: NaLa is a Java framework with a connection to a postgresql database. It can be used for generating and evaluating n-gram models. The <strong>NaLa.zip</strong> folder contains two jars of NaLa for processing json or sb3 source code files of the block-based programming language Scratch. The jar comes packed with all the required dependencies. NaLa can be run on the command line. It takes the parameters that are required for connecting to the database as parameters on startup. For guidance on how to run the jar, we added examples for calling the jar for model building, completion evaluation, and bug finding (example_configs.zip). Please find detailed information on NaLa in the README_NaLa.md. <strong>completion_ngram.zip</strong> contains the datasets as used in the n-gram completion evaluation (RQ1), json100k (for training), json10k (for evaluating) for re-running NaLa, as well as the raw experiment results completion_experiment_results, which contain the relevant output files of NaLa that we use as a basis for our evaluation. For a detailed description on what the files are used for, please find detailed information on the output of NaLa in the README_NaLa.md. <strong>bugfinding.zip</strong> contains the datasets as used in the bugfinding evaluation (RQ2), fruit_correct (for training) and fruit_schueler (for evaluating) for re-running NaLa, as well as the raw experiment results bugfinding_experiment_results as provided by NaLa without the manual classification. bugram_data_classified.zip also contains the author's classifications for the sequences that were considered in the paper. For the sake of completeness, the bugfinding_experiment_results also contain the Whisker test suite that we used, as well as the Whisker results for all the programs. <strong>transformer.zip</strong> contains the transformer_model including the Python code used to train and evaluate the transformer, the checkpoint for the best model for which the results in the paper are reported and a separate README within the archive that explains the necessary steps to run the model training and evaluation, as well as the transformer_dataset, which was used as the transformer's training and validation datasets. Further, it contains the scratch_tokenizer, which is the code used to tokenize the Scratch programs into the input format for the transformer. A separate README within the archive explains how the program can be run. Last, it contains the transformer_results as used for RQ3. <strong>scripts.zip</strong> contains the analysis scripts for generating Tables and Figures from the raw data.