Conformance Is Not Fidelity: Evaluating LLM-Constructed Feature Models-- Research Data

Conformance Is Not Fidelity: Evaluating LLM-Constructed Feature Models — Research Data This research-data archive accompanies the paper "Conformance Is Not Fidelity: Evaluating LLM-Constructed Feature Models." The study examines the distinction between structural conformance and semantic fidelity in feature models constructed by large language models. It investigates retrieval-based versus full-context grounding, iterative generation granularity, metamodel guidance, and sensitivity to evaluation conventions across two domains: model repair and model federation. Contents Corpus manifests, reference feature models, feature-to-document attribution records, and evaluation inputs. Experiment configurations, campaign matrices, available prompt snapshots, and documented protocol deviations. Generated feature models and execution records from pilot experiments, the main campaign, and a separate Astra N=1 extension. Archived recovery attempts, preserved separately from independent experimental repetitions. Derived structural, semantic, provenance, matching-sensitivity, cross-reference and statistical analyses. Paper figures, supporting tabular data, artefact manifests, and integrity checksums. The main campaign comprises 694 planned runs, including 500 completed outputs. Incomplete and failed outcomes are retained to support analysis of feasibility and conformance. Pilot experiments and the Astra extension are identified separately and must not be pooled with the main campaign. Current scope Expert evaluation is ongoing. This version does not contain expert responses or expert-evaluation analyses. Their inclusion, if any, will be documented before publication or in a subsequent version, subject to consent and privacy review. Reproducibility and limitations The archive preserves the evidence used for automated evaluation. Source-paper PDFs, processed full-text chunks, vector indexes, credentials, and private participant information are excluded. Exact prompt-template snapshots remain unavailable for 15 early pilot runs; this limitation is documented and does not affect prompt availability for the main campaign. Supporting software is maintained separately in the Feature Model Automation replication package. Archive structure The record is distributed as one preview-friendly core bundle plus optional supplementary archives, so the primary files can be previewed directly on Zenodo: feature-model-research-data-2026-10-16-core.zip — metadata, protocol, inputs, aggregated and derived analyses, paper figures, expert-study scaffolding, recovery index, artefact manifest and checksums. semantic-pairs-2026-10-16.tar.gz — raw cosine-similarity pairs per run (required only for tau-rescoring beyond the preset sweep). main-campaign-runs-2026-10-16.tar.gz — generated feature models and execution logs for the 694 planned main-campaign runs. pilots-2026-10-16.tar.gz — historical pilot experiments and pilot-characterisation analyses. Each archive preserves its directory structure. The core bundle's README explains the overall organisation; SHA-256 checksums support verification of every downloaded file. Licensing Original contributions are distributed under Creative Commons Attribution 4.0 International (CC BY 4.0). Third-party material remains subject to its applicable rights and notices; see the included licensing documentation