A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Replication Package for MODELS 2026 Research Track titled - A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL This repository provides a benchmarking pipeline for evaluating Large Language Models (LLMs) on Object Constraint Language (OCL) generation from: UML class diagrams (PlantUML format) Natural language specifications It includes: Original UML/OCL datasets Natural language specification for each UML model 3 Systematic UML transformations - identifier renaming, attribute refication, and association refication Prompting framework for LLM-based OCL generation The directory structure is as follows:├── dataset│ ├── UML│ │ ├── Airport│ │ │ ├── Airport.puml│ │ │ ├── Airport.use│ │ │ └── Airport.ocl│ │ └── EmploymentAgency│ ││ ├── Transformed_UML│ ├── specification.json│ └── transformed_specification.json │└── transformations│ ├── __init__.py│ ├── utils.py│ ├── uml_parser.py│ ├── rename_transformation.py│ ├── attribute_transformation.py│ ├── association_transformation.py│ └── runner.py├── evaluation│ ├── llm_runner.py│ ├── prompts.py│ ├── fine_tuning_LLM_for_OCL_script.ipynb│ └── config.py│└── run.py specification.json, and transformed_specification.json contain the natural language specification for each UML model Fine-Tuning fine_tuning_LLM_for_OCL_script.ipynb fine-tunes a causal LM on OCL generation using QLoRA (4-bit quantization + LoRA adapters). Dataset: fpan/text-to-ocl-from-ecore(https://huggingface.co/datasets/fpan/text-to-ocl-from-ecore), split 80/10/10 into train/val/test. Requirements: HuggingFace account with an API token, and a GPU runtime (e.g. Google Colab). To run, open the notebook, replace "HUGGINGFACE_API_TOKEN" with your token, and execute all cells. Installation 1. Download the artifact archive from: https://doi.org/10.5281/zenodo.20454636 2. Extract the archive and navigate to the extracted directory. 3. Install dependencies: pip install -r requirements.txt 4. Update evaluation/config.py with the OpenRouter API key. Evaluation Procedure The generated OCL constraints are evaluated using the USE (UML-based Specification Environment) tool. Prerequisites Install the USE tool from the official repository: - USE GitHub Repository: https://github.com/useocl/use Additional installation instructions and documentation are available in the repository. Step 1: Generate OCL Constraints To just apply the transformation on the orignal UML models: Run python run.py --mode {transformation} --task transform The following choices for the transformations could be applied: rename, attribute, association, rename_attribute, rename_association, attribute_association, full Example: python run.py --mode rename --task transform To generate OCL constraints using the benchmarking pipeline: python run.py --mode {transformation} --task llm Example: python run.py --mode rename --task llm To generate OCL expression using transformed dataset: Run python run.py --mode full --task both This produces OCL constraints generated by the selected LLM for the corresponding UML model and natural language specification. Step 2: Open the UML Model in USE For each benchmark instance: 1. Launch USE. 2. Load the corresponding .use model file located in: dataset/UML/<ModelName>/<ModelName>.use The .use file contains: - The UML class model. - The reference OCL constraints associated with the model. Step 3: Load Test Instances The repository includes object diagrams (test instances) in `.soil` format. Load a test instance into USE using either: 1. The graphical interface; or 2. The USE command line. For example: open <test_instance>.soil These instances represent valid and invalid system states against which OCL constraints can be evaluated. Step 4: Evaluate Generated OCL Constraints Replace or add the generated OCL constraint obtained from Step 1 to the loaded model. Using the loaded object instances, USE automatically evaluates whether the constraint is satisfied or violated. The evaluator should compare the behavior of the generated constraint against the expected behavior on the provided test instances. A generated OCL constraint is considered correct if it classifies the provided test instances consistently with the reference constraint. Step 5: Repeat for Transformed Models To evaluate robustness under model transformations: 1. Generate transformed UML models: python run.py --mode {transformation} --task transform 2. Generate OCL constraints for the transformed models: python run.py --mode {transformation} --task llm 3. Repeat Steps 2–4 using the transformed UML models and generated OCL constraints. Additional Documentation Detailed instructions for loading models, executing commands, and evaluating OCL constraints can be found in the USE documentation: https://github.com/useocl/use Please refer to the USE user manual for environment-specific instructions and advanced usage. Reproducing the Experimental Results To reproduce the robustness experiments reported: 1. Generate transformed datasets: python run.py --mode full --task transform 2. Generate OCL constraints: python run.py --mode full --task llm 3. Evaluate generated constraints using USE. 4. Compare the generated constraints against the reference constraints on the provided test instances. The resulting pass/fail outcomes can be used to compute the robustness metrics reported in the paper. The robustness score reported in the paper is computed as the percentage of generated OCL constraints that remain semantically equivalent to the reference constraint after transformation. Semantic equivalence is determined by executing both constraints against the provided test instances in USE and comparing their classification results. Example 1. Generate OCL for the Airport model: python run.py --mode rename --task llm Expected output: dataset/Transformed_renamed_UML/Airport/transformed_renamed_Airport.ocl 2. Open in USE tool: dataset/Transformed_renamed_UML/Airport/transformed_renamed_Airport.use 3. Load: dataset/Transformed_renamed_UML/Airport/Test_instances/*.soil Expected outcome: The generated OCL constraint should accept all valid test instances and reject all invalid test instances provided for the Airport benchmark. Supplementary Material Supplementary_Material.pdf provides real-world evidence grounding the three benchmark transformations