ms-marco-en-bge

# ms-marco-en-bge

This dataset contains the [MS MARCO](https://microsoft.github.io/msmarco/) dataset with documents similar to the query mined using [BGE-M3](https://huggingface.co/BAAI/bge-m3) and then scored by [bge-reranker-v2-m3](BAAI/bge-reranker-v2-m3).
It can be used to train a retrieval model using knowledge distillation, for example [using PyLate](https://lightonai.github.io/pylate/#knowledge-distillation).

#### `knowledge distillation`

To fine-tune a model using knowledge distillation loss we will need three distinct file:

* Datasets
```python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-en-bge",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-en-bge",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-en-bge",
"documents",
split="train",
)
```

Where:
- `train` contains three distinct columns: `['query_id', 'document_ids', 'scores']`

```python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}
```

Assert that the length of document_ids is the same as scores.

- `queries` contains two distinct columns: `['query_id', 'text']`

```python
{"query_id": 749480, "text": "what is function of magnesium in human body"}
```

- `documents` contains two distinct columns: `['document_ids', 'text']`

```python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}
```

# ms-marco-en-bge

本数据集基于[MS MARCO](https://microsoft.github.io/msmarco/)数据集构建，其中与查询语句相似的文档通过[BGE-M3](https://huggingface.co/BAAI/bge-m3)进行检索挖掘，并由[bge-reranker-v2-m3](BAAI/bge-reranker-v2-m3)完成打分。本数据集可用于基于知识蒸馏（knowledge distillation）的检索模型训练，例如借助[PyLate](https://lightonai.github.io/pylate/#knowledge-distillation)实现训练流程。

#### 知识蒸馏（knowledge distillation）

若要使用知识蒸馏损失微调模型，需使用三个独立的数据集文件：

python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-en-bge",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-en-bge",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-en-bge",
"documents",
split="train",
)

各数据集说明如下：
- `train` 数据集包含三列：`['query_id', 'document_ids', 'scores']`

python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}

需保证`document_ids`与`scores`的长度保持一致。

- `queries` 数据集包含两列：`['query_id', 'text']`

python
{"query_id": 749480, "text": "what is function of magnesium in human body"}

- `documents` 数据集包含两列：`['document_ids', 'text']`

python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}