Deriva Benchmark Data: Runs and Configurations

Benchmark data for "Taming LLM Variance in Enterprise Architecture Model Derivation from Source Code" (Grond and Proper). Deriva 0.8.10 (https://doi.org/10.5281/zenodo.23669554) derived ArchiMate models from six public repositories selected from the EA ModelSet (Cloudbased-S-BPM-WfMS, opbot, Plutus, reciptory, CAV_RECON, bigdata) with three LLMs (Claude Haiku 5.5, Mistral Small 4, GPT-6 Luna), in five runs each. Every run repeated the complete pipeline, extraction included, without an LLM cache. For comparison, a baseline run was made on the same repositories and LLMs: the LLM is given a selection of the repository's files (the README, the file tree, build manifests and the largest source files, up to 120,000 characters) and generates an ArchiMate model in one prompt; output that is not valid is returned to the LLM for repair, at most twice. Each session is exported as one bundle with the derived models (Open Group ArchiMate exchange format), per-run snapshots, every LLM call, OCEL 2.0 event logs, the step configurations at the versions used, the environment, the analysis output and a SHA-256 manifest of all files. Repositories without a licence are included by reference only: their derived models and metrics are included, LLM prompts are replaced by SHA-256 hashes, and per-run graphs are left out. dataset.zip holds the result tables, the golden scores, the reference labels per golden element (derivable, match, short reason), the repository selection and the scripts. The architects' reference models themselves are not redistributed; they are available from the EA ModelSet. v1.4: the primary golden score is symmetric (name similarity and accepted names, without the structural references, which only Deriva's elements can meet; the score with all three steps and the same-only variant stay as variants); per repository, element precision over all runs (micro, as in the pooled rows) is the primary value, next to the mean over the runs; and the release experiment is added: release-sessions.zip (its 16 sessions, complete), release-elements.csv and release-relationships.csv (change per pair of snapshots against noise, Deriva's change by cause), release.json and the scripts that ran and analysed it. The release CSVs give each difference as a count and as a share of the union of the two element sets compared. The release experiment sets the change between versions of one repository against run-to-run noise: Deriva 0.8.10 and the baseline run with one LLM (GPT-6 Luna) were each run three times on eight snapshots of Cloudbased-S-BPM-WfMS, without an LLM cache, and Deriva's change between two snapshots is traced to the files changed between them. Every v1.3 value is unchanged. v1.3: matching rule of the golden comparison documented (the primary score also counts name-similarity and structural-reference matches), a stricter same-only variant, golden elements by derivability category, and Deriva's elements and golden matches that depend on an LLM decision; every v1.2 value is unchanged (see the README version notes). Version 1.2 adds to the result tables in dataset.zip: precision pooled over all derived elements and relationships, next to the mean over the repositories; consistency pooled over the repositories, weighted by model size and as one pooled ratio; and a note on why consistency by identifier and by name differ. It leaves out the rows that listed a structural name choice as an LLM decision. Every value of version 1.1 is unchanged. Version 1.1 changed only the result tables: the LLM-dependent part of a model counts only elements decided by an LLM call (version 1.0 also counted elements of types that had an LLM call in the run), and a table attributes the relationships that differ between runs to their cause. The two benchmark session bundles are identical in all versions.