LLM-Evaluated Dataset of PubMed Abstracts for Relevance to Semantic Search in Medical Literature Reviews

This dataset contains structured insights from 16,128 PubMed abstracts, evaluated by a large language model (LLM) to assess their relevance to a research question and hypothesis exploring the impact of LLM-enhanced semantic searches on medical literature reviews. The dataset was created using the code available at https://doi.org/10.5281/zenodo.14601207, leveraging OpenAI's API Structured Output functionality to generate detailed and consistent evaluations.

The dataset was generated using PubMed titles and abstracts retrieved via the PubMed API. Details of the API endpoints and usage can be found in the referenced code repository at https://doi.org/10.5281/zenodo.14601207.

Evaluation Framework

Research Question:"How would the outcomes or insights of selected medical publications have been influenced if LLM-enhanced semantic search had been used to expand search terms and capture broader semantic matches during their literature reviews?"

Research Hypothesis:"LLM-enhanced semantic search will uncover more semantically relevant and contextually appropriate studies than traditional lexical search methods in medical literature reviews, thereby providing deeper insights and potentially altering the conclusions of previous research."

Given the inherent variability of LLM responses, future analyses using the provided code and methodology may yield different results.

Dataset Details

Fields Included:

Search Term: Manually defined search terms used in PubMed.

PMID: Unique identifier for each PubMed article (retrieved from the PubMed API).

Is Relevant: Boolean field indicating the relevance of the abstract to the research question (determined by the LLM).

Estimated Percent Relevant: A percentage estimate of the abstract's relevance (determined by the LLM).

Relevance Reason: A detailed explanation of the relevance assessment (determined by the LLM).

Article URL: Direct link to the PubMed article (retrieved from the PubMed API).

Creation Date: 2024/08/30

Methodology

PubMed article titles and abstracts were concatenated and processed by OpenAI GPT-4o via the Structured Output functionality.

The LLM was prompted with the research question, hypothesis, and the article title and abstract text to evaluate each abstract for relevance and generate structured outputs for the dataset.

Use Cases

This dataset can be used to:

Analyze the role of LLM-enhanced semantic searches in biomedical literature reviews.

Validate AI-generated relevance assessments against traditional methods.

Train and evaluate other AI models using structured biomedical data.

Acknowledgments

This dataset demonstrates the utility of LLMs in evaluating biomedical literature while highlighting the potential variability in AI-generated insights. By combining open code and data, it supports reproducibility, transparency, and collaboration in open science.