ikun1145141/bge-m3-data

---
{}
---
# Dataset Summary
This depository contains all the fine-tuning data for the [bge-m3](https://huggingface.co/BAAI/bge-m3) model, including:

| Dataset | Language |
| --------------- | :----------: |
| MS MARCO | English |
| NQ | English |
| HotpotQA | English |
| TriviaQA | English |
| SQuAD | English |
| COLIEE | English |
| PubMedQA | English |
| NLI from SimCSE | English |
| DuReader | Chinese |
| mMARCO-zh | Chinese |
| T2Ranking | Chinese |
| Law-GPT | Chinese |
| cMedQAv2 | Chinese |
| NLI-zh | Chinese |
| LeCaRDv2 | Chinese |
| Mr.TyDi | 11 languages |
| MIRACL | 16 languages |
| MLDR | 13 languages |

Note: The MLDR dataset here is the handled `train` set of the [MLDR dataset](https://huggingface.co/datasets/Shitao/MLDR).

For more details, please refer to our [paper](https://arxiv.org/pdf/2402.03216.pdf).

# Dataset Structure
Each dataset has been split into multiple files according to the tokenized length of the text (tokenizer of bge-m3, i.e. tokenizer of [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large)). For example, the MS MARCO dataset has been split into 8 files: `msmarco_len-0-500.jsonl`, `msmarco_len-500-1000.jsonl`, ..., `msmarco_len-6000-7000.jsonl`, `msmarco_len-7000-inf.jsonl`. All the files are in the `jsonl` format. Each line of the file is a json object. The following is an example of the json object:

```python
{"query": str, "pos": List[str], "neg":List[str]}
```

# Citation Information
```
@misc{bge-m3,
title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
year={2024},
eprint={2402.03216},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

# 数据集概述
本存储库包含[bge-m3](https://huggingface.co/BAAI/bge-m3)模型的全部微调数据，涵盖以下数据集：

| 数据集名称 | 语言 |
| --------------- | :----------: |
| MS MARCO | 英语 |
| NQ | 英语 |
| HotpotQA | 英语 |
| TriviaQA | 英语 |
| SQuAD | 英语 |
| COLIEE | 英语 |
| PubMedQA | 英语 |
| 基于SimCSE的自然语言推理数据集 | 英语 |
| 杜阅读（DuReader） | 中文 |
| mMARCO-zh | 中文 |
| T2Ranking | 中文 |
| 法律GPT（Law-GPT） | 中文 |
| cMedQAv2 | 中文 |
| 中文自然语言推理数据集（NLI-zh） | 中文 |
| LeCaRDv2 | 中文 |
| Mr.TyDi | 11种语言 |
| MIRACL | 16种语言 |
| MLDR | 13种语言 |

注：此处的MLDR数据集为[MLDR数据集](https://huggingface.co/datasets/Shitao/MLDR)经处理后的训练集。如需了解更多细节，请参阅我们的[论文](https://arxiv.org/pdf/2402.03216.pdf)。

# 数据集结构
每个数据集均依据文本的Token化长度（使用bge-m3的分词器，即[xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large)的分词器）拆分为多个文件。例如，MS MARCO数据集被拆分为8个文件：`msmarco_len-0-500.jsonl`、`msmarco_len-500-1000.jsonl`……`msmarco_len-6000-7000.jsonl`以及`msmarco_len-7000-inf.jsonl`。所有文件均采用`jsonl`格式，文件的每一行均为一个JSON对象。以下为该JSON对象的示例：
python
{"查询": str, "正样本": List[str], "负样本": List[str]}

# 引用信息

@misc{bge-m3,
title={BGE M3嵌入：基于自知识蒸馏的多语言、多功能、多粒度文本嵌入模型},
author={陈建旅、肖诗涛、张培天、罗坤、连德富、刘正},
year={2024},
eprint={2402.03216},
archivePrefix={arXiv},
primaryClass={cs.CL}
}