GDC-VM: The GDC Value Matching Benchmark

GDC-VM is a value matching evaluation benchmark that extends GDC-SM by providing curated matches for all attribute values, rather than only schema-level matches. The benchmark is grounded in a real-world data harmonization scenario commonly encountered in biomedical research: pooling datasets from multiple studies to increase cohort size and analytical power. This benchmark is based on the study by Li et al. (2023), in which data from ten independent studies, covering multiple patient cohorts and cancer types, were mapped to the Genomics Data Commons (GDC) data model. Annotation Process To construct the GDC-VM benchmark, we followed a multi-stage annotation process involving both automated methods and multiple human annotators. Across all annotation stages, empty target values were used to explicitly indicate cases where no suitable standardized value could be identified in the target values. Annotators were also allowed to assign multiple target values when appropriate; for example, source values such as NaN were mapped to both Not Reported and Unknown when both interpretations were considered valid. Initial automated matchingWe used BDI-Kit (version 0.9.0) to generate an initial set of value matches for each attribute across the ten datasets. Primary human reviewA human annotator reviewed all automatically generated matches and corrected them when necessary, producing a revised list of value matches. Independent secondary reviewsFor each dataset, two additional human annotators independently reviewed the matches and generated their own lists of value matches. Some of these matches overlapped with those from the primary annotator. Disagreement analysisWe compared the three independently generated lists and identified cases without absolute agreement (i.e., where the three annotators did not selected the same value match). Overall inter-annotator agreement exceeded 95%. Consensus resolutionAll three annotators jointly reviewed the disagreement cases and reached a final consensus, updating their choices when necessary. The resulting consensus matches constitute the final ground truth of the GDC-VM benchmark. Statistics All statistics reported correspond to counts of unique elements, including: source attributes, target attributes, attribute-level matches, source values, target values, and value-level matches. Source Attributes: distinct attributes in the original study dataset. Target Attributes: distinct attributes in the GDC data model. Attribute Matches: schema-level matches between source and target attributes. Source Values: unique values observed in the source attributes. Target Values: unique standardized values in the corresponding GDC attributes. Value Matches: curated value-level correspondences forming the ground truth. Across the ten datasets, the benchmark covers a diverse range of schema sizes and value distributions, reflecting realistic heterogeneity in biomedical data harmonization tasks. Source datasets contain between 5 and 29 attributes, with 8 to 41 attribute-level matches per dataset. At the value level, datasets range from 117 to 524 source values and from 114 to 423 target values, resulting in 182 to 840 curated value matches per dataset. Dataset # Source Attributes # Target Attributes # Attribute Matches # Source Values # Target Values # Value Matches Cao 29 41 41 524 399 840 Clark 13 19 20 229 223 427 Dou 19 28 28 225 219 419 Gillette 14 18 19 279 307 386 Huang 19 29 32 171 146 352 Krug 5 8 8 133 138 182 McDermott 14 19 19 140 122 216 Satpathy 12 17 20 117 114 223 Vasaikar 13 16 16 232 235 266 Wang 20 23 24 428 423 591 How to Use Datasets The data consist of ten CSV files: Cao.csv, Clark.csv, Dou.csv, Gillette.csv, Huang.csv, Krug.csv, McDermott.csv, Satpathy.csv, Vasaikar.csv and Wang.csv. Each CSV file contains matches between source attributes and values from an original dataset and the corresponding attributes and values in the GDC target schema. The files share a common structure with columns such as source_attribute, target_attribute, source_value, and target_value. GDC schema The file gdc_schema.json contains the GDC target schema (version 3.1.0), including target attributes and their allowed values (with associated metadata and descriptions as provided by GDC). We also provide a flattened tabular version of the schema in gdc_schema_flattened.csv. Loading the data The script load_data.py provides a minimal example of how to load: (i) all CSV mapping files into a single pandas DataFrame; and (ii) the GDC target schema into a Python dictionary. This script is intended as a lightweight starting point for users who want to explore, analyze, or validate the matches. Due to licensing restrictions, the original source datasets cannot be redistributed in this repository. However, a script to download the original source datasets is provided here. Citation For citation purposes, refer to this dataset as described below: Lopez, R., Wu, E., Keegan, S., Liu, W., Liu, Y., Koutras, C., Pinto, D., Marini, P., Fenyo, D., & Freire, J. (2026). GDC-VM: The GDC Value Matching Benchmark [Data set]. Zenodo. https://doi.org/10.5281/zenodo.18557064