A Gold Standard Corpus for Activity Information (GoSCAI)

A Gold Standard Corpus for Activity Information Dataset Title: A Gold Standard Corpus for Activity Information (GoSCAI) Dataset Curators: The Epidemiology & Biostatistics Section of the NIH Clinical Center Rehabilitation Medicine Department Dataset Version: 1.0 (May 16, 2025) Dataset Citation and DOI: NIH CC RMD Epidemiology & Biostatistics Section. (2025). A Gold Standard Corpus for Activity Information (GoSCAI) [Data set]. Zenodo. doi: 10.5281/zenodo.15528545 EXECUTIVE SUMMARY This data statement is for a gold standard corpus of de-identified clinical notes that have been annotated for human functioning information based on the framework of the WHO's International Classification of Functioning, Disability and Health (ICF). The corpus includes 484 notes from a single institution within the United States written in English in a clinical setting. This dataset was curated for the purpose of training natural language processing models to automatically identify, extract, and classify information on human functioning at the whole-person, or activity, level. CURATION RATIONALE This dataset is curated to be a publicly available resource for the development and evaluation of methods for the automatic extraction and classification of activity-level functioning information as defined in the ICF. The goals of data curation are to 1) create a corpus of a size that can be manually deidentified and annotated, 2) maximize the density and diversity of functioning information of interest, and 3) allow public dissemination of the data. LANGUAGE VARIETIES Language Region: en-US Prose Description: English as written by native and bilingual English speakers in a clinical setting LANGUAGE USER DEMOGRAPHIC The language users represented in this dataset are medical and clinical professionals who work in a research hospital setting. These individuals hold professional degrees corresponding to their respective specialties. Specific demographic characteristics of the language users such as age, gender, or race/ethnicity were not collected. ANNOTATOR DEMOGRAPHIC The annotator group consisted of five people, 33 to 76 years old, including four females and one male. Socioeconomically, they came from the middle and upper-middle income classes. Regarding first language, three annotators had English as their first language, one had Chinese, and one had Spanish. Proficiency in English, the language of the data being annotated, was native for three of the annotators and bilingual for the other two. The annotation team included clinical rehabilitation domain experts with backgrounds in occupational therapy, physical therapy, and individuals with public health and data science expertise. Prior to annotation, all annotators were trained on the specific annotation process using established guidelines for the given domain, and annotators were required to achieve a specified proficiency level prior to annotating notes in this corpus. LINGUISTIC SITUATION AND TEXT CHARACTERISTICS The notes in the dataset were written as part of clinical care within a U.S. research hospital between May 2008 and November 2019. These notes were written by health professionals asynchronously following the patient encounter to document the interaction and support continuity of care. The intended audience of these notes were clinicians involved in the patients' care. The included notes come from nine disciplines - neuropsychology, occupational therapy, physical medicine (physiatry), physical therapy, psychiatry, recreational therapy, social work, speech language pathology, and vocational rehabilitation. The notes were curated to support research on natural language processing for functioning information between 2018 and 2024. PREPROCESSING AND DATA FORMATTING The final corpus was derived from a set of clinical notes extracted from the hospital electronic medical record (EMR) for the purpose of clinical research. The original data include character-based digital content originally. We work in ASCII 8 or UNICODE encoding, and therefore part of our pre-processing includes running encoding detection and transformation from encodings such as Windows-1252 or ISO-8859 format to our preferred format. On the larger corpus, we applied sampling to match our curation rationale. Given the resource constraints of manual annotation, we set out to create a dataset of 500 clinical notes, which would exclude notes over 10,000 characters in length. To promote density and diversity, we used five note characteristics as sampling criteria. We used the text length as expressed in number of characters. Next, we considered the discipline group as derived from note type metadata and describes which discipline a note originated from: occupational and vocational therapy (OT/VOC), physical therapy (PT), recreation therapy (RT), speech and language pathology (SLP), social work (SW), or miscellaneous (MISC, including psychiatry, neurology and physiatry). These disciplines were selected for collecting the larger corpus because their notes are likely to include functioning information. Existing information extraction tools were used to obtain annotation counts in four areas of functioning and provided a note’s annotation count, annotation density (annotation count divided by text length), and domain count (number of domains with at least 1 annotation). We used stratified sampling across the 6 discipline groups to ensure discipline diversity in the corpus. Because of low availability, 50 notes were sampled from SLP with relaxed criteria, and 90 notes each from the 5 other discipline groups with stricter criteria. Sampled SLP notes were those with the highest annotation density that had an annotation count of at least 5 and a domain count of at least 2. Other notes were sampled by highest annotation count and lowest text length, with a minimum annotation count of 15 and minimum domain count of 3. The notes in the resulting sample included certain types of PHI and PII. To prepare for public dissemination, all sensitive or potentially identifying information was manually annotated in the notes and replaced with substituted content to ensure readability and enough context needed for machine learning without exposing any sensitive information. This de-identification effort was manually reviewed to ensure no PII or PHI exposure and correct any resulting readability issues. Notes about pediatric patients were excluded. No intent was made to sample multiple notes from the same patient. No metadata is provided to group notes other than by note type, discipline, or discipline group. The dataset is not organized beyond the provided metadata, but publications about models trained on this dataset should include information on the train/test splits used. All notes were sentence-segmented and tokenized using the spaCy en_core_web_lg model with additional rules for sentence segmentation customized to the dataset. Notes are stored in an XML format readable by the GATE annotation software (https://gate.ac.uk/family/developer.html), which stores annotations separately in annotation sets. CAPTURE QUALITY As the clinical notes were extracted directly from the EMR in text format, the capture quality was determined to be high. The clinical notes did not have to be converted from other data formats, which means this dataset is free from noise introduced by conversion processes such as optical character recognition. LIMITATIONS Because of the effort required to manually deidentify and annotate notes, this corpus is limited in terms of size and representation. The curation decisions skewed note selection towards specific disciplines and note types to increase the likelihood of encountering information on functioning. Some subtypes of functioning occur infrequently in the data, or not at all. The deidentification of notes was done in a manner to preserve natural language as it would occur in the notes, but some information is lost, e.g. on rare diseases. METADATA Information on the manual annotation process is provided in the annotation guidelines for each of the four domains: - Communication & Cognition (https://zenodo.org/records/13910167) - Mobility (https://zenodo.org/records/11074838) - Self-Care & Domestic Life (SCDL) (https://zenodo.org/records/11210183) - Interpersonal Interactions & Relationships (IPIR) (https://zenodo.org/records/13774684) Inter-annotator agreement was established on development datasets described in the annotation guidelines prior to the annotation of this gold standard corpus. The gold standard corpus consists of 484 documents, which include 35,147 sentences in total. The distribution of annotated information is provided in the table below. Domain Number of Annotated Sentences % of All Sentences Mean Number of Annotated Sentences per Document Communication & Cognition 6033 17.2% 12.5 Mobility 2282 6.5% 5.0 SCDL 1472 4.2% 3.0 IPIR 2954 8.4% 6.1 DISCLOSURES AND ETHICAL REVIEW This project was supported by the intramural research program of the National Institutes of Health. The NIH Office of Human Subjects Research Protections (OHRP) determined that research with these data did not meet the definition of human subjects research pursuant to 45 CFR 46 and OHRP guidance. DISTRIBUTION This dataset will be shared with researchers who have been granted permission to use the data the Epidemiology & Biostatistics Section, Rehabilitation Medicine Department, NIH Clinical Center community published on Zenodo. The dataset will have restricted access to those researchers who request permission to use the data and agree to the NIH Clinical Center's terms of use. Note that the NIH negotiates agreements with entities (e.g., universities) rather than individuals. To request access, send an email to [e-mail] and include a description of the intended research project that would be conducted with the dataset. MAINTENANCE This dataset is hosted on Zenodo with Restricted Access. The creators of this dataset can be contacted by sending an email to [e-mail]. Updates may be performed to address issues found through internal review or external reports, but there is no set schedule for such updates to be performed or communicated. ## About this document A data statement is a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software. This data statement was written based on the template for the Data Statements Version 3 schema. The template was prepared by Angelina McMillan-Major and Emily M. Bender and can be found at https://techpolicylab.uw.edu/data-statements/ and was updated from the community [Version 1 Markdown template by Leon Dercyznski] (https://gist.github.com/leondz/b3a53bb807a301424e3762787a04a5da).