ms-marco-fr-bge

# ms-marco-fr-bge

This dataset contains the [french version](https://huggingface.co/datasets/unicamp-dl/mmarco) of [MS MARCO](https://microsoft.github.io/msmarco/) dataset with documents similar to the query mined using [BGE-M3](https://huggingface.co/BAAI/bge-m3) and then scored by [bge-reranker-v2-m3](BAAI/bge-reranker-v2-m3). It can be used to train a retrieval model using knowledge distillation.

#### `knowledge distillation`

To fine-tune a model using knowledge distillation loss we will need three distinct file:

* Datasets
```python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-fr-bge",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-fr-bge",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-fr-bge",
"documents",
split="train",
)
```

Where:
- `train` contains three distinct columns: `['query_id', 'document_ids', 'scores']`

```python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}
```

Assert that the length of document_ids is the same as scores.

- `queries` contains two distinct columns: `['query_id', 'text']`

```python
{"query_id": 749480, "text": "what is function of magnesium in human body"}
```

- `documents` contains two distinct columns: `['document_ids', 'text']`

```python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}
```

# ms-marco-fr-bge

该数据集为[MS MARCO](https://microsoft.github.io/msmarco/)的[法语版本](https://huggingface.co/datasets/unicamp-dl/mmarco)，其中与查询相似的文档通过[BGE-M3](https://huggingface.co/BAAI/bge-m3)挖掘得到，并由[bge-reranker-v2-m3](BAAI/bge-reranker-v2-m3)进行打分。该数据集可用于基于知识蒸馏（knowledge distillation）训练检索模型。

#### 知识蒸馏（knowledge distillation）

若使用知识蒸馏损失微调模型，则需要三个独立的数据集文件：
* Datasets
python
from datasets import load_dataset

train = load_dataset(
"lightonai/ms-marco-fr-bge",
"train",
split="train",
)

queries = load_dataset(
"lightonai/ms-marco-fr-bge",
"queries",
split="train",
)

documents = load_dataset(
"lightonai/ms-marco-fr-bge",
"documents",
split="train",
)

各数据集说明如下：
- 训练集（train）包含三列字段：`['query_id', 'document_ids', 'scores']`

python
{
"query_id": 54528,
"document_ids": [
6862419,
335116,
339186,
7509316,
7361291,
7416534,
5789936,
5645247,
],
"scores": [
0.4546215673141326,
0.6575686537173476,
0.26825184192900203,
0.5256195579370395,
0.879939718687207,
0.7894968184862693,
0.6450100468854655,
0.5823844608171467,
],
}

需保证`document_ids`与`scores`的长度一致。

- 查询集（queries）包含两列字段：`['query_id', 'text']`

python
{"query_id": 749480, "text": "what is function of magnesium in human body"}

- 文档集（documents）包含两列字段：`['document_ids', 'text']`

python
{
"document_id": 136062,
"text": "2. Also called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.lso called tan .a fundamental trigonometric function that, in a right triangle, is expressed as the ratio of the side opposite an acute angle to the side adjacent to that angle. 3. in immediate physical contact; touching; abutting. 4. a. touching at a single point, as a tangent in relation to a curve or surface.",
}