Large Language Models for Software Models: An Empirical Evaluation in Game Software Engineering

Replication Package

This repository is the replication package for the paper "Large Language Models for Software Models: An Empirical Evaluation in Game Software Engineering" (under review). It contains all materials needed to reproduce the controlled empirical evaluation of seven state-of-the-art LLMs on Unity Visual Scripting (UVS) manipulation tasks, as well as the statistical analysis of the results.

Study Overview

Seven LLMs are evaluated on eight UVS manipulation tasks spanning four difficulty levels (Easy, Medium, Hard, and Ultra). Each task–model combination is executed five times, yielding 280 executions in total. Correctness is measured with Precision, Recall, and F1-score computed by a rule-based validation pipeline. Statistical analysis applies Kruskal-Wallis tests with Dunn post-hoc comparisons (Bonferroni-corrected), reporting η² as the omnibus effect size and rank-biserial r for pairwise comparisons.

Models evaluated:

Anthropic Claude 4.5 Sonnet

Google Gemini 2.5 Pro

OpenAI GPT-5.2

DeepSeek R1 (70b)

Google Gemma3 (27b)

OpenAI GPT-OSS (120b)

Meta LLaMA 3.2 (3b)

Tasks by difficulty level:

EASY:

01_easy_rotate_naming

02_easy_rotate_broken_wire

MEDIUM:

03_medium_movement_speed_dt_broken_wire

04_medium_movement_axis_diag_normalize

HARD:

05_hard_rotate_invert_and_fix_names_plus_movement_axes

06_hard_repair_broken_refs_restore_movement_rotation

ULTRA:

07_ultra_generate_rotation

08_ultra_generate_movement_rotation

Repository Structure

├── README.md ← This file
│
├── 01_Experiment/ ← Experiment runner, tasks, outputs, and validator
│ ├── tasks/ ← Benchmark task definitions (8 tasks)
│ ├── outputs/ ← Pre-collected LLM outputs (280 executions)
│ └── uvs_validator/ ← Rule-based validation
│ ├── parsers/ ← UVS asset and pack parsers
│ ├── tokenizers/ ← Unit tokenizer
│ ├── utils/ ← Graph analysis utilities
│ └── validators/ ← Rule checks (basic and advanced)
│
└── 02_StatisticalAnalysis/ ← R analysis scripts and data
├── executions.csv ← Scored executions (280 rows; input to R)
├── kruskal_wallis.R ← Full statistical analysis script
└── kruskal_results.txt ← Output produced by kruskal_wallis.R

License

This replication package is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/). You are free to share and adapt the materials for any purpose, provided appropriate credit is given.