OSCAT-Exec: Execution-based benchmark, model outputs and analysis code for evaluating small local LLMs generating IEC 61131-3 Structured Text

Replication package for the article "Compiling is not correct: An execution-based study of small, locally deployable language models generating IEC 61131-3 Structured Text" (Raszka, Grycz, Vilamová; submitted to Empirical Software Engineering). It contains the OSCAT-Exec benchmark (197 tasks derived from the OSCAT BASIC library with prompts, generated test inputs and reference outputs), the RuSTy-based differential-testing harness, all responses and evaluations of eleven open-weight LLMs (0.5–14.8B parameters, GGUF Q4_K_M, greedy decoding) including compiler-feedback repair and dialect-normalisation re-evaluation, CPU throughput measurements, mutation analysis of the oracle, specification-adequacy and root-cause annotations, and scripts that reproduce every table and figure. See README_replication.md for instructions and tool versions.