Optimizing the input for automated SNOMED CT mapping of clinical variables - Dataset and Evaluation

In this work, we investigated the effect of variable name (VN) formulation and descriptiveness on the accuracy and efficiency of their mapping to medical terminology “Systematized Nomenclature of Medicine Clinical Terms” (SNOMED CT) with the “Medical Concept Annotation Toolkit” (MedCAT). We extracted 763 VNs from the German registry of paediatric deep brain stimulation in patients with childhood-onset dystonia (GEPESTIM) hosted locally with “Research electronic data capture” (REDCap). Subseqeuntly, we created three VN versions using different REDCap metadata items for MedCAT-based SNOMED CT mapping. The metadata items comprise the REDCap end-user label (EUL), form name (FN) and Branching logic (BL). A fourth VN version was created manually. The input creation can be traced in file "MedCAT_input.csv". The mapping was evaluated by two evaluators based on the number and quality of identified SNOMED CT concepts, using manual scoring from -3 to +3 to assess concept accuracy while ensuring a blind evaluation process. A score of 0 means that no content-carrying concepts were mapped, while negative scores indicate the mapping of incorrect concepts. The higher the proportion of incorrect concepts, the lower the score. Positive Scores were assigned if only correct concepts were mapped. The score increases with increasing completeness of medical concept coverage. If the opposite of the correct concepts was mapped, we assigned a score of -2. The MedCAT Output annotated by our evaluations can be found in files "*_Concept_Mapping_Evaluation.csv". Overview MedCAT_input.csv - REDCap metadata plus VN versions created based on that data baseline_Concept_Mapping_Evaluation.csv - MedCAT output on VN version "baseline" annotated with evaluations manual_Concept_Mapping_Evaluation.csv - MedCAT output on VN version "manual" annotated with evaluations withFormName_Concept_Mapping_Evaluation.csv - MedCAT output on VN version "withFormName" annotated with evaluations withFormNameAndBL_Concept_Mapping_Evaluation.csv - MedCAT output on VN version "withFormNameAndBL" annotated with evaluations