Transition and Emission Probabilities for UMLS Semantic Type Codes in CORD-19 Data

Text from CORD-19 dataset (April 2020) was segmented into sentences and annotated with entity span markers using SciSpacy (english, medium), then linked to UMLS concepts using the SciSpacy + UMLS integration (UMLSKnowledgeBase). This linking is noisy, i.e., a span can link with multiple UMLS concepts. We filtered for sentences where there is no duplicate linkage, and reduced them to sequences of UMLS semantic type codes, then computed Transition and Emission probabilities for consecutive semantic code pairs across the corpus of selected sentences.