Why Do LLMs Fail at OCL Generation? A Graph Reasoning Perspective

Replication Package - Why Do LLMs Fail at OCL Generation? A Graph Reasoning Perspective This replication package contains the dataset, scripts, and prompts needed to replicate the empirical study on why Large Language Models (LLMs) fail at generating Object Constraint Language (OCL) constraints. Directory Structure replication_package/├── dataset/ # Dataset directory│ ├── UML/ # UML class diagrams (PlantUML format)│ │ ├── Airport/│ │ ├── EURental/│ │ ├── EmploymentAgency/│ │ ├── HealthRecord/│ │ ├── ISP/│ │ ├── InvoicingOrders/│ │ ├── Mortgage/│ │ ├── Person/│ │ ├── QUDV/│ │ ├── Royal&Loyal/│ │ ├── Tournament/│ │ ├── Train/│ │ └── Vehicle/│ ├── UML_Structured/ # Structured UML models with ordering│ ├── generated_ocl/ # Generated OCL constraints from experiments│ │ ├── Airport/│ │ ├── EURental/│ │ └── ... (per model)│ ├── prompts.json # LLM prompts used for generation│ ├── specifications.json # Ground truth OCL specifications│ ├── feature_dataset*.csv # Feature datasets for analysis│ ├── OCL_metrics.csv # OCL correctness metrics│ ├── pivot_dataset.py # Utility script for pivoting data│ └── reorder_uml.py # Utility script for reordering UML│├── src/ # Source code for experiments│ ├── generate.py # Main script for OCL generation via LLM│ ├── OCL-generation-V-LLM-baseline.py # Baseline generation script│ ├── main.py # Entry point for running experiments│ ├── structural_complexity_analysis.py # Analysis of structural complexity│ ├── final_structural_complexity_analysis.py # Final analysis│ ├── merge_features.py # Script to merge feature datasets│ ├── plot_class_count_correctness_regression.py # Plotting script│ ├── features/ # Feature extraction module│ │ └── feature_builder.py│ ├── modeling/ # Modeling module│ │ ├── uml_model.py│ │ └── logistic_model.py│ ├── parser/ # UML parser module│ │ └── uml_parser.py│ ├── similarity/ # Similarity calculation module│ │ └── similarity_engine.py│ ├── utils/ # Utility functions│ │ └── io_utils.py│ ├── visualization/ # Visualization scripts│ │ ├── plot_analysis.py│ │ ├── plot_class_count_bins_correctness.py│ │ └── structural_complexity_verification.py│ └── results/ # Analysis results│ └── structural_complexity/│├── results/ # Experimental results│ ├── feature_dataset*.csv # Merged feature datasets│ ├── lexical_similarity/ # Lexical similarity results│ ├── structural_complexity/ # Structural complexity results│ ├── structural_complexity_report.txt│ └── mixed_effects_report.txt│└── requirements.txt # Python dependencies Prerequisites Python 3.8+ OpenRouter API key (for LLM queries) Python Dependencies Install the required dependencies: pip install -r requirements.txt The following packages are required: pandas>=2.0.0 numpy>=1.24.0 scikit-learn>=1.3.0 statsmodels>=0.14.0 matplotlib>=3.7.0 seaborn>=0.12.0 Dataset Description UML Models (`dataset/UML/`) The dataset contains 14 UML class diagrams in PlantUML format, each representing a domain-specific model: | Model | Domain ||-------|--------|| Airport | Flight and passenger management || EURental | Car rental system || EmploymentAgency | Employee and project management || HealthRecord | Medical records || ISP | Internet service provider || InvoicingOrders | Order and invoicing || Mortgage | Mortgage lending || Person | Person and family relationships || QUDV | Quantities and units || Royal&Loyal | Loyalty program || Tournament | Sports tournament || Train | Train and wagon management || Vehicle | Vehicle ownership | Specifications (`dataset/specifications.json`) Contains 115 ground truth OCL constraints with their natural language specifications. Each specification includes:- Natural language description- Ground truth OCL expression Prompts (`dataset/prompts.json`) Contains the prompts used for LLM-based OCL generation:- generation: System and user prompts for full generation- completion: Completion-style prompts Test Instances Each UML model has associated test instances (.soil files) in the Test_instances/ subdirectory. These are used for OCL correctness validation. Running the Experiments Step 1: Generate OCL Constraints To generate OCL constraints using LLMs, run: cd replication_package/srcpython generate.py Configuration: Edit the API_KEY variable in generate.py to use your OpenRouter API key Modify the MODELS list to test different LLMs Configure UML_CONFIGS to control UML presentation order Step 2: Run Analysis Scripts After generating OCL, run the analysis scripts: # Structural complexity analysispython structural_complexity_analysis.py # Final structural complexity analysispython final_structural_complexity_analysis.py # Merge feature datasetspython merge_features.py Step 3: Generate Visualizations # Plot class count vs correctness regressionpython plot_class_count_correctness_regression.py # Plot class count binspython plot_class_count_bins_correctness.py Key Scripts | Script | Purpose ||--------|---------|| `generate.py` | Main script to generate OCL using LLMs via OpenRouter API || `main.py` | Entry point for running the full experiment pipeline || `structural_complexity_analysis.py` | Analyzes structural complexity of generated OCL || `feature_builder.py` | Extracts features from UML models and OCL constraints || `similarity_engine.py` | Calculates similarity between generated and reference OCL | Output The experiments generate: 1. Generated OCL (dataset/generated_oc/) - OCL constraints generated by each LLM2. Feature Datasets (results/) - CSV files with extracted features3. Analysis Reports (results/) - Text files with statistical analysis4. Visualizations - Plots showing the relationship between model characteristics and correctness Notes The `generate.py` script requires an OpenRouter API key. You can obtain one at https://openrouter.ai/ Some scripts may require significant computation time depending on the number of models and specifications