UML-to-Java Model Transformation with Large Language Models

This repository contains the replication package for the research work:

An empirical evaluation of Large Language Models (LLMs) for UML-to-Java model transformation in a zero-shot setting.

The study investigates whether modern Large Language Models can serve as out-of-the-box model transformers, translating UML class diagrams serialized in XMI/Ecore into structurally correct Java code, without explicit transformation rules, in-context examples, or interactive correction.

Building on the experimental framework originally proposed by Kazai et al., this work extends the evaluation in scale, rigor, and analysis depth, using:

116 UML class diagram models,

three state-of-the-art LLMs (GPT-5.2, DeepSeek v3, Gemini 2.5),

a fixed zero-shot prompt, and

a two-iteration regeneration protocol.

The repository implements the entire experimental pipeline, including oracle generation, LLM execution, automated comparison, error filtering, and result aggregation.

Research Context and Goal

Model transformation is a cornerstone of Model-Driven Engineering (MDE), but traditional rule-based transformations (e.g., ATL, QVT) introduce significant accidental complexity and maintenance overhead.

This research evaluates whether LLMs can:

lower the entry barrier to MDE,

act as direct UML-to-Java transformers, and

preserve structural fidelity across increasing model complexity.

The central research question addressed by this repository is:

To what extent can pretrained LLMs perform UML-to-Java model transformation correctly and consistently, without explicit transformation rules, and how does this capability degrade with model complexity?

Experimental Design (Implemented in This Repository)

The pipeline follows a constructive and experimental research design, consisting of the following phases:

Dataset collection and screening

Oracle Java generation

Zero-shot LLM-based transformation

Automated comparison and manual filtering

Iterative regeneration (maximum two iterations)

Result aggregation and error analysis

Iteration Policy (Important)

Although the original pipeline supported up to three iterations, empirical analysis showed that:

The third iteration produced no additional error-free transformations.

Therefore, this repository intentionally supports only two iterations, which:

capture all observed improvements,

avoid artificial inflation of results, and

keep the evaluation controlled and comparable.

All scripts, folder structures, and result reporting reflect this two-iteration design choice.

Prerequisites

The pipeline was executed and validated on Windows, using the following tools:

Windows OS

AutoHotKey (v1.1 and v2.0)

Beyond Compare

Visual Studio Code

Language Support for Java™ by Red Hat (v1.28.1)

Python

Google Chrome

ChatGPT (web interface)

DeepSeek (web interface)

Gemini (web interface)

The use of AutoHotKey requires that LLMs be accessed via a browser and that user interaction not be interrupted during execution.

Installation and Configuration

Java Formatting Configuration

Enable automatic Java formatting in VS Code:

Settings → Command Palette → Preferences: Configure Language Specific Settings

→ Java → Text Editor → Formatting → Format On Save

Path Configuration

You must specify local executable paths in the following files:

VS Code path

formatJava.ahk (lines 4, 11)

formatJava_SecondIteration.ahk (line 3)

Python path

firstIteration.bat

secondIteration.bat

specificExecution.bat

(Line numbers are documented inside each file.)

Beyond Compare

Place a shortcut to Beyond Compare in the repository root directory.

Prompt Used for LLM Transformation

All transformations use the same fixed zero-shot prompt:

The following is an XMI file based on a UML diagram.

Please write corresponding Java code without adding any explanatory text.

Do not implement getters/setters and do not add comments.

Generate classes as if they were in separate files.

Start and end the response with [[STARTEND]] as a Java comment.

No examples, hints, or oracle excerpts are provided.

Execution

Demo Run

To verify correct installation, run:

specificExecution.bat

and specify:

demo

After execution, results are stored in:

./allXMI/works/demo/

Folder contents:

oracle/ – Java reference generated via Modelio or ATL+Python

gpt/ – XMI input and LLM-generated Java output

comparison/ – Beyond Compare reports and error logs

Iteration Workflow (Two Iterations Only)

First Iteration

Executed via firstIteration.bat

Produces initial LLM output

Compares against oracle

Classifies results as error-free or erroneous

Second Iteration

Executed via secondIteration.bat

Re-runs only failed cases

Uses the same prompt (no corrections injected)

Produces final cumulative results

There is no third iteration in this repository, by design.

Repository Structure

allXMI/

└── works/

        └── done/

                └── "UML model_s_name"/

                         ├── 2nd ite/ (if needed)

                         ├── result/

                         ├── oracle/

                         └── comparison/

Key subfolders for each model:

oracle/ – Reference Java code generated by Modelio or ATL+Python

result/ – LLM outputs

comparison/ – Comparison reports and error logs

Evaluation Method

Evaluation combines automated tooling and manual inspection, following established practice:

Modelio / ATL+Python → oracle generation

Python scripts → file normalization

AutoHotKey → automation

Beyond Compare → structural differencing

Manual filtering → removal of cosmetic differences

A transformation is classified as error-free only if no structural discrepancies remain after filtering.

Replication Notes

The dataset, prompts, and iteration logic are fixed

Results are reproducible if the file structure is preserved

For the correct setup, refer to the demo folder as a template

Built With

AutoHotKey

Beyond Compare

Modelio 4.1

Python

Visual Studio Code

Java Language Support (Red Hat)

ChatGPT (web interface)

Gemini (web interface)

DeepSeek (web interface)