UML-Gold Corpus

UML-Gold Corpus: A Consolidated Ground-Truth Dataset for Requirements Engineering and Conceptual Modeling Dataset Description The UML-Gold Corpus (gold_standard.jsonl) is a highly granular, human-annotated dataset designed to advance research at the intersection of Natural Language Processing (NLP) and Software Engineering (SE). Specifically, it targets the tasks of Joint Named Entity Recognition (NER) and Relation Extraction (RE) to automate the generation of Unified Modeling Language (UML) diagrams (such as Use Case and Class Diagrams) directly from raw software requirements and user stories. The corpus consists of diverse software specifications sourced from curated software engineering repositories (including MENDELEY and DOSSPRE) across multiple domain projects (e.g., g13-planningpoker, g19-alfred, and more). Each requirement is annotated down to precise character-level spans to represent real-world software abstractions, their relationships, and data properties. Inter-Rater Reliability & Curation Provenance This dataset is the final, consolidated output of a rigorous Inter-Rater Reliability (IRR) Agreement framework. To ensure the highest degree of annotation fidelity and scientific validity, the data went through a multi-stage consensus pipeline: Multi-Expert Annotation: Three independent Subject Matter Experts (SMEs) independently annotated the raw textual requirements within a Doccano instance using a strict, predefined UML Metamodel Ontology. Statistical Alignment Assessment: Pairwise alignment matrices were built dynamically across character offsets to calculate chance-corrected consensus metrics via Cohen's Kappa and multi-rater Fleiss' Kappa. Automated Structural & Schema Auditing: The annotations were vetted using deterministic validation scripts checking label conformity, entity boundaries, and relation target integrity. Submissions failing semantic criteria were returned for adjudication. Conflict Resolution & Consolidation: Inter-annotator discrepancies, overlapping boundary choices, and edge variations were reviewed and resolved systematically by the experts to build this unified, absolute ground-truth dataset—the UML-Gold Corpus. Technical Specifications & Format File Name: gold_standard.jsonl Format: JSON Lines (.jsonl), UTF-8 encoded. Each row represents a standalone JSON object mapping an independent requirement statement. Nested Entity Support: The schema explicitly supports Nested Named Entity Recognition. Because software terminology is compositional, entity spans may reside within larger target actions (e.g., the noun token text "account" labeled as a CLASS can live structurally inside the larger verb phrase span "create an account", which is typed as a USE_CASE). Spans are mapped via half-open, zero-indexed intervals: [start_offset, end_offset). Field-Level Schema Reference id (Integer): Global tracking index within the annotation repository. sent_id (Integer): Relational positional index of the requirement text segment. text (String): Raw, untokenized requirements statement context. type (String): Architectural classification of the requirement (e.g., Functional, Functionality). source (String): Academic or industrial data repository origin (e.g., MENDELEY, DOSSPRE). project_id (String): Unique token identifier grouping texts belonging to the same software project environment. entities (Array): Array of extracted structural semantic entities. Each item contains: id (Integer): Unique text-bound entity instance index. label (String): The ontological class assignment. text (String): Literal text substring captured by the offsets. start_offset (Integer): Character index boundary beginning the text span. end_offset (Integer): Character index boundary ending the text span. relations (Array): Graph edge configurations linking structural nodes. Each item contains: id (Integer): Unique relation instance tracker token. type (String): Semantic edge dependency relationship classification. from_id (Integer): The originating/head entity ID token. to_id (Integer): The destination/tail entity ID token. Ontological Taxonomy (UML Metamodel Axioms) The annotations follow a strict, domain-specific conceptual modeling metamodel defined in the system taxonomy rules. 1. Named Entity Labels (Nodes) ACTOR: Represents a user profile, external system, or system environment role that interacts with the target software application boundary (e.g., "moderator", "MedicalCaregiver"). SYSTEM_BOUNDARY: Defines the software scope, application framework, platform system, or module execution perimeter (e.g., "application", "ALFRED", "system"). USE_CASE: Represents the functional goals, runtime actions, or system behaviors executed by actors to achieve value targets (e.g., "create an account", "determine the user's breathing frequency"). CLASS: Highlights abstract concepts, domain objects, entities, or structural data modules that exist internally within the logic framework (e.g., "account", "modules"). OPERATION: Identifies specific system processes, functions, or programmatic methods belonging to classes or domain blocks. Typed Data Attributes: Properties or data types associated with structural classes or system actors, strongly typed by field data values: STRING_ATTRIBUTE (e.g., "password", "username", "name") FLOAT_ATTRIBUTE (e.g., "breathing frequency") INTEGER_ATTRIBUTE / LONG_ATTRIBUTE DATE_ATTRIBUTE BOOLEAN_ATTRIBUTE BLOB_ATTRIBUTE 2. Relation Types (Edges) Relationships are governed by rigid structural metamodel directional rules: PERFORMS: Maps operational capabilities. Connects an ACTOR (head) directly to a USE_CASE (tail), indicating deployment behavior. OWNS: Connects an organizational structural node (either an ACTOR or CLASS) to an associated data property classification (* _ATTRIBUTE), denoting possession. CONTAINS: Connects a SYSTEM_BOUNDARY to a functional USE_CASE or internal CLASS, explicitly modeling structural containment limits. PART_OF: Models compositional, aggregation, or sub-component topology connections between two internal systemic layers (e.g., a CLASS or component being structurally part of a SYSTEM_BOUNDARY). Target Research Use Cases The UML-Gold Corpus provides a high-quality benchmark for training and evaluating data-driven software engineering tools. Potential use cases include: Joint Entity and Relation Extraction (JERE): Benchmarking state-of-the-art neural architectures (e.g., Transformer-based token classification models, LLM fine-tuning regimes) on complex domain structures. Automated Conceptual Modeling: Engineering tools capable of passing natural language code statements or documentation texts and generating syntactically valid UML Use Case and Class diagrams automatically. Nested Named Entity Recognition: Exploring spatial alignment heuristics, span parsing algorithms, and attention mechanisms handling multi-word layered expressions. Keywords Requirements Engineering UML Modeling Named Entity Recognition Relation Extraction Inter-Rater Reliability Software Engineering NLP Ground Truth Corpus User Stories Conceptual Modeling