pubmed-h5n1

# PubMed H5N1 Articles

_Current as of January 5, 2025_

This dataset is metadata (id, publication date, title, link) from PubMed articles related to H5N1. It was created using [paperetl](https://github.com/neuml/paperetl) and the [PubMed Baseline](https://pubmed.ncbi.nlm.nih.gov/download/).

The 37 million articles were filtered to match either of the following criteria.

- MeSH code = [D053124](https://meshb-prev.nlm.nih.gov/record/ui?ui=D053124)
- Keyword of `H5N1` in either the `title` or `abstract`

## Retrieve article abstracts

The full article abstracts can be retrieved via the [PubMed API](https://www.nlm.nih.gov/dataguide/eutilities/utilities.html#efetch). This method accepts batches of PubMed IDs.

Alternatively, the dataset can be recreated using the following steps and loading the abstracts into the dataset (see step 5).

## Download and build

The following steps recreate this dataset.

1. Create the following directories and files

```bash
mkdir -p pubmed/config pubmed/data

echo "D053124" > pubmed/config/codes
echo "H5N1" > pubmed/config/keywords
```

2. Install `paperetl` and download `PubMed Baseline + Updates` into `pubmed/data`.

```bash
pip install paperetl datasets
```

3. Parse the PubMed dataset into article metadata

```bash
python -m paperetl.file pubmed/data pubmed/articles pubmed/config
```

4. Export to dataset

```python
from datasets import Dataset

ds = Dataset.from_sql(
("SELECT id id, published published, title title, reference reference FROM articles "
"ORDER BY published DESC"),
f"sqlite:///pubmed/articles/articles.sqlite"
)
ds.to_csv(f"pubmed-h5n1/articles.csv")
```

5. _Optional_ Export to dataset with all fields

paperetl parses all metadata and article abstracts. If you'd like to create a local dataset with the abstracts, run the following instead of step 4.

```python
import sqlite3
import uuid

from datasets import Dataset

class Export:
def __init__(self, dbfile):
# Load database
self.connection = sqlite3.connect(dbfile)
self.connection.row_factory = sqlite3.Row

def __call__(self):
# Create cursors
cursor1 = self.connection.cursor()
cursor2 = self.connection.cursor()

# Get article metadata
cursor1.execute("SELECT * FROM articles ORDER BY id")
for row in cursor1:
# Get abstract text
cursor2.execute(
"SELECT text FROM sections WHERE article = ? and name != 'TITLE' ORDER BY id",
[row[0]]
)
abstract = " ".join(r["text"] for r in cursor2)

# Combine into single record and yield
row = {**row, **{"abstract": abstract}}
yield {k.lower(): v for k, v in row.items()}

def __reduce__(self):
return (pickle, (str(uuid.uuid4()),))

def pickle(self, *args, **kwargs):
raise AssertionError("Generator pickling workaround")

# Path to database
export = Export("pubmed/articles/articles.sqlite")
ds = Dataset.from_generator(export)
ds = ds.sort("published", reverse=True)
ds.to_csv("pubmed-h5n1-full/articles.csv")
```

# PubMed H5N1 文献数据集

_更新至2025年1月5日_

本数据集收录与H5N1相关的PubMed文献元数据，包含文献ID、发表日期、标题及链接。本数据集通过[paperetl](https://github.com/neuml/paperetl)与[PubMed基础数据集（PubMed Baseline）](https://pubmed.ncbi.nlm.nih.gov/download/)构建生成。

本次共筛选3700万篇PubMed文献，符合以下任一条件即可纳入数据集：
- 医学主题词表（Medical Subject Headings，MeSH）编码为[D053124](https://meshb-prev.nlm.nih.gov/record/ui?ui=D053124)
- 标题或摘要中包含关键词`H5N1`

## 文献摘要获取

完整的文献摘要可通过[PubMed API（PubMed应用程序编程接口）](https://www.nlm.nih.gov/dataguide/eutilities/utilities.html#efetch)获取，该接口支持批量传入PubMed文献ID。

此外，也可通过以下步骤重新构建数据集并将摘要纳入其中（详见步骤5）。

## 数据集下载与构建

以下为重新构建本数据集的完整步骤：

1. 创建如下目录与配置文件：
bash
mkdir -p pubmed/config pubmed/data

echo "D053124" > pubmed/config/codes
echo "H5N1" > pubmed/config/keywords

2. 安装`paperetl`与`datasets`Python库，并将`PubMed基础数据集及更新包`下载至`pubmed/data`目录下：
bash
pip install paperetl datasets

3. 将PubMed数据集解析为文献元数据：
bash
python -m paperetl.file pubmed/data pubmed/articles pubmed/config

4. 导出为基础数据集：
python
from datasets import Dataset

ds = Dataset.from_sql(
("SELECT id id, published published, title title, reference reference FROM articles "
"ORDER BY published DESC"),
f"sqlite:///pubmed/articles/articles.sqlite"
)
ds.to_csv(f"pubmed-h5n1/articles.csv")

5. **可选**：导出包含全字段的完整数据集

paperetl会解析所有元数据与文献摘要。若需创建包含完整摘要的本地数据集，可执行以下代码替代步骤4：
python
import sqlite3
import uuid

from datasets import Dataset

class Export:
def __init__(self, dbfile):
# Load database
self.connection = sqlite3.connect(dbfile)
self.connection.row_factory = sqlite3.Row

def __call__(self):
# Create cursors
cursor1 = self.connection.cursor()
cursor2 = self.connection.cursor()

# Get article metadata
cursor1.execute("SELECT * FROM articles ORDER BY id")
for row in cursor1:
# Get abstract text
cursor2.execute(
"SELECT text FROM sections WHERE article = ? and name != 'TITLE' ORDER BY id",
[row[0]]
)
abstract = " ".join(r["text"] for r in cursor2)

# Combine into single record and yield
row = {**row, **{"abstract": abstract}}
yield {k.lower(): v for k, v in row.items()}

def __reduce__(self):
return (pickle, (str(uuid.uuid4()),))

def pickle(self, *args, **kwargs):
raise AssertionError("Generator pickling workaround")

# Path to database
export = Export("pubmed/articles/articles.sqlite")
ds = Dataset.from_generator(export)
ds = ds.sort("published", reverse=True)
ds.to_csv("pubmed-h5n1-full/articles.csv")