Benchmarking LLM for MTBE

FILES OVERVIEW:

The experiment involves multiple transformation tasks and configurations across five LLMs. All input/output files follow a structured naming convention:

[filename]_[model]_[n]_examples.[filetype]These files contain the output model generated by each LLM under a given scenario and number of example pairs.

[filename].[filetype]These files are the example model to be input.

mappingrules[inputType][outputType].texThese files store the mapping rules written in abstract machine-readable xmi format.

[inputType/outputType]metamodel.texThese files contain metamodels for either desired input or output models.

Filename Example
Description

VehicleDefinitions_gpt_2_examples
Output by GPT-4.5 for SysML-to-AAS with 2 example pairs

Port Examples.aas
An AAS model used as part of an example pair for SysML-to-AAS

mappingrulesSysMLAAS.tex
Mapping rules for scenario SysML-to-AAS

AASmetamodel.tex
AAS metamodel used for scenario SysML-to-AAS

KEY POINTERS WHEN RE-DOING THE EXPERIMENTS:

Prompt Composition: Prompts must follow the format:Prompt = Mapping Rules (M) + n Example Pairs (En) + Metamodels (MM)

Semantic Completeness: Ignore small formatting issues. Prioritize semantic and structural correctness.

Model-Specific Behavior: Be aware that LLMs like GPT-4.5 are sensitive to prompt length and example order.

Manual Review: Since some outputs include hallucinations or missing fields, results must be manually reviewed for meaningful validation.

STEPS TO REPLICATE THE PROCESS:

1. Prompt Generation

Prepare prompt for each scenario (RDBMS-to-UML, UML-to-Java, SysML-to-AAS) and configuration (1pairs, 2pairs, 4pairs), each including:

Textual mapping rules (M)

Example pairs (E)

Metamodel descriptions (MM)

New source model for transformation

You should have one prompt per configuration per scenario.

2. Running LLMs

For each of the five tested LLMs:

Copy the entire prompt text into the LLM’s web interface (ChatGPT, Claude, Gemini, etc.).

Paste manually—do not modify the formatting or insert extra instructions.

Specify the task in the system prompt if available, e.g.,"You are a transformation assistant that learns from example models and generates new target models."

Models Tested:

ChatGPT-4.5

DeepSeek V3

DeepHermes 3 LLaMA 3 8B

QwQ 32B

OlympicCoder 32B

Repeat the process for:

3 scenarios (RDBMS-to-UML, UML-to-Java, SysML-to-AAS) 

3 configurations (1, 2, 4 example pairs) = 9 prompts per model

3. Save the Output

After the model responds, copy the generated model into a .[outputType] file named using the format:

[filename]_[model]_[n]pairs.[outputType]

Make no edits to the output at this point—preserve the raw generation.

3. Manual Evaluation

Compare LLM-generated models against ground-truth.

Use two validation metrics:

% Correct: Cr / (LOC + Miss)

% Weighted Success: (Cr − InCr + Ad − 2×Miss) / LOC

Use Excel/Google Sheets for structured tracking:

Columns: Model, Scenario, #Pairs, LOC, Cr, InCr, Ad, Miss, %Correct, %WSuccess

4. Repeat for All Models and Configurations

Total combinations to run and evaluate:

3 Scenarios × 3 Configs × 5 LLMs = 45 runs

Each output must be individually reviewed and scored.