Large Language Models are Zero-Shot Medical Knowledge Encoders: Benchmark

This repository contains the benchmark aiming at evaluating the quality of SNOMED-CT embeddings. It cinvolves two different benchmarks.

Hierarchical Similarity Benchmark contains 10151 rows and 7 columns:- fsn: The Fully Specified Name of the evaluated concept, i.e. the label of the SNOMED concept.- sctid: The SNOMED ID of the evaluated concept.- close_fsn: The Fully Specified Name of the concept that is supposed to be an immediate parent, or an immediate child (depending on the distance type, child or parent).- close_sctid: The SNOMED ID of the closer concept.- far_fsn: The Fully Specified Name of the concept that is supposed to be farther.- far_sctid: The SNOMED ID of the farther concept.- distance_type: This indicates whether this sample is generated downward (parentdistance) or upward (childdistance). Also, X_depth_Y means the current node is of depth X, and is Y hops away from the farther concept.

Semantic Composition Benchmark contains 1042 rows and 4 rows:- id_node: The ID of the child concept- fsn_node: The Fully Specified Name of the child- parents_ids: The IDs of both parents that are the child's components- parents_fsn: The Fully Specified Name of the parents