ms-marco-en-bge-gemma

# ms-marco-en-bge

This dataset contains the [MS MARCO](https://microsoft.github.io/msmarco/) dataset with negatives mined using ColBERT and then scored by [bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma).
It can be used to train a retrieval model using knowledge distillation, for example [using PyLate](https://lightonai.github.io/pylate/#knowledge-distillation).

#### `knowledge distillation`

To fine-tune a model using knowledge distillation loss we will need three distinct file:

* Datasets
```python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-en-gemma",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-en-gemma",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-en-gemma",
"documents",
split="train",
)
```

Where:
- `train` contains three distinct columns: `['query_id', 'document_ids', 'scores']`

```python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}
```

Assert that the length of document_ids is the same as scores.

- `queries` contains two distinct columns: `['query_id', 'text']`

```python
{"query_id": 749480, "text": "what is function of magnesium in human body"}
```

- `documents` contains two distinct columns: `['document_ids', 'text']`

```python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}
```

# ms-marco-en-bge
本数据集基于[MS MARCO](https://microsoft.github.io/msmarco/)数据集构建，通过ColBERT挖掘负样本，并由[bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)对负样本进行打分。该数据集可用于基于知识蒸馏（knowledge distillation）训练检索模型，例如可借助[PyLate](https://lightonai.github.io/pylate/#knowledge-distillation)完成相关训练。

#### 知识蒸馏（knowledge distillation）
若要使用知识蒸馏损失对模型进行微调，需要使用三个独立的数据集文件：
* 数据集加载代码：
python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-en-gemma",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-en-gemma",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-en-gemma",
"documents",
split="train",
)

各数据集说明如下：
- 训练集（train）包含三列数据：`['query_id', 'document_ids', 'scores']`
其数据格式示例如下：
python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}

需保证`document_ids`与`scores`的长度一致。
- 查询集（queries）包含两列数据：`['query_id', 'text']`
其数据格式示例如下：
python
{"query_id": 749480, "text": "what is function of magnesium in human body"}

- 文档集（documents）包含两列数据：`['document_ids', 'text']`
其数据格式示例如下：
python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}