A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Replication Package for MODELS 2026 Research Track titled - A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL This repository provides a benchmarking pipeline for evaluating Large Language Models (LLMs) on Object Constraint Language (OCL) generation from: UML class diagrams (PlantUML format) Natural language specifications It includes: Original UML/OCL datasets Natural language specification for each UML model 3 Systematic UML transformations - identifier renaming, attribute refication, and association refication Prompting framework for LLM-based OCL generation The directory structure is as follows:├── dataset│ ├── UML│ │ ├── Airport│ │ │ ├── Airport.puml│ │ │ ├── Airport.use│ │ │ └── Airport.ocl│ │ └── EmploymentAgency│ ││ ├── Transformed_UML│ ├── specification.json│ └── transformed_specification.json │└── transformations│ ├── __init__.py│ ├── utils.py│ ├── uml_parser.py│ ├── rename_transformation.py│ ├── attribute_transformation.py│ ├── association_transformation.py│ └── runner.py├── evaluation│ ├── llm_runner.py│ ├── prompts.py│ ├── fine_tuning_LLM_for_OCL_script.ipynb│ └── config.py│└── run.py specification.json, and transformed_specification.json contain the natural language specification for each UML modelTo transform a dataset: Run python run.py --mode {transformation} --task transformExample: python run.py --mode rename --task transformTo generate OCL expressions using the LLM:Run python run.py --mode {transformation} --task llmExample: python run.py --mode rename --task llm To generate OCL expression using transformed dataset: Run python run.py --mode full --task both Fine-Tuningfine_tuning_LLM_for_OCL_script.ipynb fine-tunes a causal LM on OCL generation using QLoRA (4-bit quantization + LoRA adapters). Dataset: fpan/text-to-ocl-from-ecore(https://huggingface.co/datasets/fpan/text-to-ocl-from-ecore), split 80/10/10 into train/val/test. Requirements: HuggingFace account with an API token, and a GPU runtime (e.g. Google Colab). To run, open the notebook, replace "HUGGINGFACE_API_TOKEN" with your token, and execute all cells. Supplementary MaterialSupplementary_Material.pdf provides real-world evidence grounding the three benchmark transformations