DOME Copilot - Data Files (V1)

DOME Copilot Experiment: Metadata, Supplementary Data, and Evaluation Source Files Dataset Overview This dataset supports the research on "DOME Copilot", an AI-assisted workflow for curating machine learning experiment metadata from scientific publications. It aggregates the machine-readable output of the AI models, the ground truth human reviews, and the processed supplementary data files associated with the analyzed publications. Archive Content: DOME_Copilot_Data_Package.zip The main data package is organized into the following directories and files: 1. Metadata Index DOME_Copilot_Data_Package_Metadata.csv: A comprehensive index mapping every PMCID (PubMed Central ID - PMCID) in the dataset to the directories/datasets where its data can be found. Use this to quickly locate resources for a specific paper. 2. DOME Registry Database - Human Annotations DOME_Registry_Human_Reviews_258_20260205.json: The core dataset used to create the benchmark dataset, these are publications containing expert human-curated metadata annotations for the DOME Registry available papers. 3. Copilot Output (AI Generated) Copilot_Processed_Datasets_JSON/: Contains the structured JSON metadata generated by the DOME Copilot pipeline. 222 Benchmark Cohort: Copilot_222_v0_Processed_2025-12-04 and its updated metadata. Copilot_222_v2_Processed_2026-03-02 and its updated metadata. 1012 Experimental Cohorts: Positive (ML papers): Copilot_1012_v2_Pos_Processed_2026-03-02, v0 outputs, and their updated metadata. Negative (Non-ML papers): Copilot_1012_v2_Neg_Processed_2026-03-02 and updated metadata. These directories represent different model iterations (v0 & v2) and contain the structured output data extracted directly from the publications. Note: we did not have a negative dataset for v0 experiments, so no v0 available for these. 4. Main Text & Supplementary PDF Materials This archive contains the raw PDF full text and supplementary material downloaded from PMC for the study datasets, which serves as input data for the DOME Copilot analyses (Section 3). DOME_Registry_222_PMCID_PDFs_PMC_Full_Text_and_Supplementary/: Full text and Supplementary files for the 222 benchmark dataset articles of the DOME Registry (Benchmark Set - Has Available Human Annotations). Positive_1012_PDFs_PMCID_Full_Text_and_Supplementary/: Full text and Supplementary files for the 1012 "Positive" articles (Machine Learning relevant - Positive Set). Negative_1012_PDFs_PMCID_Full_Text_and_Supplementary/: Full text and Supplementary files for the 1012 "Negative" articles (Negative Set / Non-ML). 5. Evaluation Data Human_30_Evaluation_Source_JSONs_Human_and_Copilot_Including_PDFs/: The specific source files used for the deep-dive comparative evaluation (Human vs. AI) on a random subset but diverse of 30 papers. Contains the Copilot JSONs, Human JSONs, and original PDFs used in the evaluation interface. Notes Matching: The DOME_Copilot_Data_Package_Metadata.csv file is the key to linking entries across these folders. Not all papers have supplementary data. Privacy: User-specific data (profiles, emails) has been excluded. PDFs: Full-text and supplementary PDFs are included in this bulk archive sourced from PMC through APIs in compliance with licensing for research. Identifiers - PMCID: PMCIDs are the common identifiers in use across folders. Data processing scripts & info: https://github.com/gavinf97/DOME-Copilot-Data-Analysis