A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Replication Package for MODELS 2026 Research Track titled - A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL This repository provides a benchmarking pipeline for evaluating Large Language Models (LLMs) on Object Constraint Language (OCL) generation from: UML class diagrams (PlantUML format) Natural language specifications It includes: Original UML/OCL datasets Natural language specification for each UML model 3 Systematic UML transformations - identifier renaming, attribute refication, and association refication Prompting framework for LLM-based OCL generation The directory structure is as follows:├── dataset│ ├── UML│ │ ├── Airport│ │ │ ├── Airport.puml│ │ │ ├── Airport.use│ │ │ └── Airport.ocl│ │ └── EmploymentAgency│ ││ ├── Transformed_UML│ ├── specification.json│ └── transformed_specification.json │└── transformations│ ├── __init__.py│ ├── utils.py│ ├── uml_parser.py│ ├── rename_transformation.py│ ├── attribute_transformation.py│ ├── association_transformation.py│ └── runner.py├── evaluation│ ├── llm_runner.py│ ├── prompts.py│ └── config.py│└── run.py specification.json, and transformed_specification.json contain the natural language specification for each UML modelTo transform a dataset: Run python run.py --mode {transformation} --task transformExample: python run.py --mode rename --task transformTo generate OCL expressions using the LLM:Run python run.py --mode {transformation} --task llmExample: python run.py --mode rename --task llm To generate OCL expression using transformed dataset: Run python run.py --mode full --task both