遇见数据集

Deriva Benchmark Data: Runs and Configurations

收藏
Zenodo2026-11-02 更新2026-11-05 收录
官方服务:

资源简介:

Benchmark data for "Taming LLM Variance in Enterprise Architecture Model Derivation from Source Code" (Grond and Proper). Deriva 0.8.10 (https://doi.org/10.5281/zenodo.23669554) derived ArchiMate models from six public repositories selected from the EA ModelSet (Cloudbased-S-BPM-WfMS, opbot, Plutus, reciptory, CAV_RECON, bigdata) with three LLMs (Claude Haiku 5.5, Mistral Small 4, GPT-6 Luna), in five runs each. Every run repeated the complete pipeline, extraction included, without an LLM cache. For comparison, a baseline run was made on the same repositories and LLMs: the LLM is given a selection of the repository's files (the README, the file tree, build manifests and the largest source files, up to 120,000 characters) and generates an ArchiMate model in one prompt; output that is not valid is returned to the LLM for repair, at most twice. A release experiment sets the change between versions of one repository against run-to-run noise: both systems with one LLM (GPT-6 Luna), three runs on each of eight snapshots of Cloudbased-S-BPM-WfMS, without an LLM cache. Each session is exported as one bundle with the derived models (Open Group ArchiMate exchange format), per-run snapshots, every LLM call, OCEL 2.0 event logs, the step configurations at the versions used, the environment, the analysis output and a SHA-256 manifest of all files. Repositories without a licence are included by reference only: their derived models and metrics are included, LLM prompts are replaced by SHA-256 hashes, and per-run graphs are left out. dataset.zip holds the result tables, the golden scores, the labels per golden element, the repository selection and the scripts. The architects' reference models themselves are not redistributed; they are available from the EA ModelSet. v1.7: the golden labels were checked again, more consistently, and scored anew; the first labels and their scores stay in v1.6. The label files say whether a hand or a rule set each label. README and selection.md are shortened. The session bundles are unchanged, and so are the consistency, stability, cost and release results. Earlier version notes are in the earlier versions' records.

提供机构:
Zenodo
创建时间:
2026-11-02
二维码
社区交流群
二维码
科研交流群
商业服务