BioGraphletQA: A complex biomedical Knowledge Graph Question Answering dataset

This document provides a detailed description of the data files associated with the paper "BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasets".

This repository contains the full dataset, intermediate outputs from our generation pipeline, and the rephrased subsets used for downstream task evaluation.

1. Generation Templates

This file contains the raw graphlets extracted from the OREGANO knowledge graph, which serve as the factual basis for the entire generation process.

File: templates.jsonl

Total Entries: 269,574

Description: Each line is a JSON object representing a single graphlet (a small, connected subgraph).

Schema: templates.jsonl

Key
Type
Descriptor

id
Integer
A unique identifier for the graphlet.

nodes
List[String]
A list of node IDs from the OREGANO KG (e.g., "COMPOUND:78").

edges
List[List[Int]]
An edge list representing the connections between nodes, using indices from the nodes list.

graphlet_id
Integer
An ID corresponding to one of the 29 unique graphlet shapes.

edge_labels
List[String]
The relation type for each edge.

node_names
List[String]
The human-readable names corresponding to the node IDs.

graphlet_text
String
A formatted string listing the nodes and their names for easy readability.

num_nodes
Integer
The total number of nodes in the graphlet.

2. Raw Generated QA Pairs

This file contains the direct output from the initial LLM generation stage, after basic cleaning (JSON parsing validation and Z-score outlier removal).

File: cleaned_dataset.jsonl - from _llm_outs/DATASET/

Total Entries: 264,373

Description: Each line is a JSON object containing the original graphlet information plus the full text output from the LLM, which includes its reasoning, self-reflection, and the final generated question-answer pair.

Schema: cleaned_dataset.jsonl

This file includes all fields from templates.jsonl plus the following:

Key
Type
Description

text
String
The complete, raw text output from the LLM, including its analysis and reflection steps.

question
String
The final generated question.

answer
String
The final generated answer.

json_parsed
Boolean
A flag indicating that the LLM's output was successfully parsed as JSON.

3. Filtered Datasets

These files are the output of the LLM-as-a-judge filtering stage. The dataset is split into pairs that were accepted (deemed high-quality) and those that were rejected.

File (Accepted): filtered_dataset.jsonl - from _llm_outs/FILTER_DATASET/

Total Entries (Accepted): 119,856

File (Rejected): rejected_dataset.jsonl - from _llm_outs/FILTER_DATASET/

Total Entries (Rejected): 127,416

Note: An additional 17,101 pairs that were not parseable as JSON are excluded from these files.

Schema: Filtered Files

These files include all fields from cleaned_dataset.jsonl plus the following, which contains the reasoning from the LLM filter:

Key
Type
Description

filtering_text
String
The full text output from the LLM filter, including its reasoning, its generated answer for comparison, and the final boolean judgments.

4. Supporting Document Retrieval

This section describes the two files related to enriching QA pairs with documents from PubMed.

4.1. BM25 Retrieved Documents (Input for Annotation)

File: shuffled_data_qa.jsonl - from _retrieval

Description: This file contains the top 10 PubMed abstracts retrieved by the BM25 algorithm for each QA pair. This serves as the input for the LLM annotation step, using the 2025 PubMed index.

Schema: shuffled_data_qa.jsonl

Key
Type
Description

id
Integer
The unique ID of the QA pair.

query_text
String
The concatenated question and answer used as a query for retrieval.

bm25
List[Dict]
A list of the top 10 retrieved documents. Each dictionary contains the PubMed id and the full text.

4.2. LLM-Annotated Documents (Final Output)

File: retrieval_annotated_data.jsonl from _retrieval/

Description: This file is the output of the annotation process. It contains the QA pairs, the original retrieved documents, and the LLM's relevance judgment and extracted snippets for each document.

Schema: retrieval_annotated_data.jsonl

This file includes all fields from the input file plus processed_documents.

Key
Type
Description

processed_documents
List[Dict]
The LLM's annotation. Each dictionary contains the PubMed id, a boolean relevant flag, and a list of snippets (strings) extracted by the LLM.

5. Rephrased Datasets for Downstream Tasks

These files contain a subset of 20,000 QA pairs that have been automatically rephrased to match the format of specific downstream benchmarks.

File (MedQA): _rephrase/20_000_qwen3_MedQA_rephrase.json

File (PubMedQA): _rephrase/20_000_qwen3_PubMedQA_rephrase.json

Note: These files are saved with a .jsonl extension for compatibility but are formatted as single, large JSON objects.

Schema: Rephrased Files

Each file is a single JSON object where keys are the id of the original QA pair.

Key
Type
Description

id
Integer
The unique ID of the original QA pair.

qa
String
The original question and answer, concatenated.

medQA_similar
List[String]
A list of five example questions from the target dataset, used to guide the LLM's rephrasing.

snippets
List[String]
The supporting text snippets from the retrieval stage.

medQA_rephrase
Dict
A dictionary containing the newly generated multiple-choice question (question), five options, reasoning, and the correct option letter (cop).