Benchmarking LLM over roundtrip evolution of Ecore and SQL

FILES OVERVIEW This repository evaluates round-trip Model Transformation-by-Example (MTBE) between Ecore (S) and SQL-XMI (T). Each experiment starts from a paired model (S, T), applies a single operator delta Δ to one side, then performs: Inverse step: Δ(T) → S′ Forward step: S′ → T′ We validate by comparing S′ vs Δ(S) and T′ vs Δ(T). KEY POINTERS WHEN RE-DOING THE EXPERIMENTS Prompt Composition (same formula):Prompt = Example Pair(s) (E) + Δ-variant input (IM) + Stage Task Semantic Completeness: Ignore minor formatting/whitespace. Prioritize structural and referential correctness (classes/tables, attributes/columns, types, multiplicities/keys, containment vs reference, FKs). Model-Specific Behavior: LLMs are sensitive to prompt length and example order. Keep the same example pair(s) across both stages of a run. Manual Review: Outputs may contain hallucinations or omissions. Always perform loadability and semantic checks before scoring. STEPS TO REPLICATE THE PROCESS 1. Prompt Generation Prepare two prompts per (pairId, op, k): A) Inverse (Δ(T) → S′)Include (in this order): Example Pair(s) (Eₙ) — S_[pairId].ecore + T_[pairId].sql.xmi (k = 1, 2, or 4) Input Artifact — Delta_[op]_T_[pairId].sql.xmi Task - “Provide only the Ecore/XMI serialization of S’, metamodel-conformant. If ∆(T ) contains elements not covered by the example mapping, briefly flag them in XML comments, but still produce a valid S’.” B) Forward (S′ → T′)Same structure, but the Input Artifact is the S′ you just produced.Task - “Generate only SQL-XMI for T′ (no prose/comments). Ensure metamodel conformance and Δ-awareness.” Keep E fixed across both stages to isolate round-trip effects. 2. Running LLMs For each of the four tested LLMs: GPT-5 DeepSeek V3 Qwen-3-Next-80B-A3B Gemini 2.5 Execution tips: Optional system prompt:“You are a transformation assistant that learns from example models and produces structurally valid models only.” 3. Manual Evaluation Compare: Inverse: *_inverse_*.ecore (S′) vs Delta_[op]_S_[pairId].ecore (Δ(S)) Forward: *_forward_*_sql.xmi (T′) vs Delta_[op]_T_[pairId]_sql.xmi (Δ(T)) Record per stage in a sheet with columns: LOC, Cr, InCr, Ad, Miss, %Correct, %WSuccess, DeltaDetected, Loadable Metrics (same style ): % Correct = Cr / (LOC + Miss − Ad) % Weighted Success = (Cr−InCr−Miss−Ad[harmful]−2[∆ undetected] / LOC Delta checks: S′ ≡ Δ(S) and T′ ≡ Δ(T) (after canonical whitespace/ID normalization) Semantic/structural equivalence (entities, properties, types, multiplicities, FKs, containment vs reference), even if IDs/order differ from the original files. Loadability sanity: Ensure XML namespaces (xmi:version, xmlns:*, nsURI) match the metamodel declarations. 4. Repeat for All Combinations Let: B = number of base pairs (here 10) D = number of delta operators (here 10) L = number of LLMs (here 4) Then you’ll have: Runs: B × D × L, each with two saved outputs (inverse + forward) Stage evaluations: 2 × B × D × L (here 800)