lumees/ms-marco-tr-hard-negatives

---
language:
- tr
tags:
- search
- information-retrieval
- sentence-transformers
- msmarco
- hard-negatives
license: mit
task_categories:
- text-retrieval
source_datasets:
- parsak/msmarco-tr
size_categories:
- 100K<n<1M
---

# MS MARCO TR - Hard Negatives Dataset

## Dataset Description

* **Source Repository:** Derived from [parsak/msmarco-tr](https://huggingface.co/datasets/parsak/msmarco-tr).
* **Language:** Turkish (`tr`)
* **Task:** Semantic Search, Dense Retrieval, Embedding Training
* **Size:** ~500k - 1M Training Triplets (Depending on processed queries)
* **Authors:** Lumees AI, Hasan Kurşun, Kerem Berkay Yanık
* **Year:** 2025
* **Website:** [lumees.io](https://lumees.io)

### Dataset Summary
This dataset contains **Hard Negatives** specifically mined for the Turkish MS MARCO dataset. It is designed for training or fine-tuning sentence embedding models (e.g., SBERT) for Turkish Information Retrieval tasks.

[Image of vector space diagram showing query positive hard negative and random negative]

Unlike standard random negatives, these "hard" negatives are passages that share **high semantic similarity** (high vector similarity) with the query but are **not the correct answer**. Training on this data forces the model to learn subtle semantic distinctions, significantly improving retrieval performance.

### Creation Process
1. **Source Data:** Training queries and passages were taken from the `parsak/msmarco-tr` dataset (Machine translated MS MARCO).
2. **Mining Model:** The [emrecan/bert-base-turkish-cased-mean-nli-stsb-tr](https://huggingface.co/emrecan/bert-base-turkish-cased-mean-nli-stsb-tr) model was used.
3. **Method:**
* **Encoding:** All queries and passages were converted into dense vectors using the mining model.
* **Retrieval:** For each query, the top 100 similar passages were retrieved using **Faiss** (Inner Product).
* **Filtering:** The true positive (correct answer) was removed from the results.
* **Safety Threshold:** Passages with a similarity score higher than **0.98** were discarded to prevent "False Negatives" (correct answers accidentally labeled as negative).
* **Selection:** From the remaining candidates, the top **10** passages with the highest scores were selected as Hard Negatives.

---

## Data Structure

### Data Examples
Each line in the `.jsonl` file is a valid JSON object representing a training example.

```json
{
"query": "Manhattan projesinin başarısının hemen etkisi neydi?",
"pos": "Manhattan Projesi ve atom bombası, İkinci Dünya Savaşı'nın sona ermesine yardımcı oldu...",
"negatives": [
"Manhattan Projesi, II. Dünya Savaşı sırasında ilk atom bombasını geliştirmek için...",
"Proje, nükleer silah üretimi üzerine odaklanmıştı...",
"..."
],
"scores": [
0.874,
0.852,
"..."
]
}
````

### Data Fields

* **`query`** (string): The search query.
* **`pos`** (string): The true positive passage (correct answer).
* **`negatives`** (list of strings): A list of 10 passages that are semantically close to the query but incorrect. Sorted by similarity (highest to lowest).
* **`scores`** (list of floats): Cosine similarity scores corresponding to the passages in the `negatives` list. Useful for margin-based filtering or weighted loss functions during training.

-----

## Usage Guide

### Loading the Dataset (Python)

You can load this dataset using the Hugging Face `datasets` library or standard JSON line reading methods.

```python
from datasets import load_dataset

# If uploaded to Hugging Face
ds = load_dataset("lumees/msmarco-tr-hard-negatives", split="train")

# If loading from a local file
ds = load_dataset("json", data_files="msmarco_tr_hard_negatives_final.jsonl", split="train")

print(ds[0])
```

### Training with Sentence Transformers

This dataset is optimized for loss functions like `MultipleNegativesRankingLoss` or `InfoNCE`.

```python
from sentence_transformers import InputExample

train_examples = []
for row in ds:
# Structure: [Query, Positive, Negative1, Negative2, ...]
texts = [row['query'], row['pos']] + row['negatives']
train_examples.append(InputExample(texts=texts))

# Note: Ensure the Loss function you are using supports multiple negatives per example.
```

-----

## Limitations & Bias

1. **Translation Errors:** The original `parsak/msmarco-tr` dataset was created via machine translation from English. Therefore, some Turkish expressions may not be natural or may contain translation errors.
2. **False Negatives:** Despite the `0.98` similarity filter, there is a possibility that some passages selected as "negatives" are actually correct answers that were not labeled in the original dataset.
3. **Model Bias:** The negatives were mined using the `emrecan/bert-base-turkish` model. The dataset naturally reflects the biases and semantic understanding of this base model.

## Citation

If you use this dataset, please cite Lumees AI, the original MS MARCO authors, and the Turkish translation source as follows:

```bibtex
@misc{lumees_msmarco_hn_2025,
author = {Lumees AI and Kurşun, Hasan and Yanık, Kerem Berkay},
title = {MS MARCO TR - Hard Negatives Dataset},
year = {2025},
howpublished = {\url{[https://lumees.io](https://lumees.io)}},
}

@article{bajaj2016ms,
title={MS MARCO: A Human Generated Machine Reading Comprehension Dataset},
author={Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and others},
journal={arXiv preprint arXiv:1611.09268},
year={2016}
}

@misc{parsak_msmarco_tr,
author = {Parsak},
title = {MS MARCO Turkish Translation},
year = {2023},
publisher = {Hugging Face},
journal = {Hugging Face Hub},
howpublished = {\url{[https://huggingface.co/datasets/parsak/msmarco-tr](https://huggingface.co/datasets/parsak/msmarco-tr)}}
}
```

language:
- 土耳其语（tr）
tags:
- 搜索
- 信息检索
- 句子转换器（sentence-transformers）
- MS MARCO
- 难负样本（hard negatives）
license: MIT协议
task_categories:
- 文本检索
source_datasets:
- parsak/msmarco-tr
size_categories:
- 10万 < 样本数 < 100万
---

# MS MARCO TR - 难负样本数据集

## 数据集说明

* **源数据仓库：** 派生自 [parsak/msmarco-tr](https://huggingface.co/datasets/parsak/msmarco-tr)。
* **语言：** 土耳其语（`tr`）
* **任务：** 语义搜索、稠密检索、嵌入模型训练
* **规模：** 约50万至100万条训练三元组（依处理的查询数量而定）
* **作者：** Lumees AI、Hasan Kurşun、Kerem Berkay Yanık
* **发布年份：** 2025
* **官方网站：** [lumees.io](https://lumees.io)

### 数据集概览

本数据集专为土耳其语版MS MARCO数据集挖掘生成**难负样本（hard negatives）**，旨在针对土耳其语信息检索任务，训练或微调句子嵌入模型（如SBERT）。

[向量空间示意图：展示查询、正样本、难负样本与随机负样本]

与标准随机负样本不同，此类“难”负样本为与查询语义相似度极高（向量相似度高）但并非正确答案的段落。基于此数据集训练可迫使模型学习细微的语义差异，显著提升检索性能。

### 数据集构建流程
1. **源数据：** 训练查询与段落取自`parsak/msmarco-tr`数据集（MS MARCO的机器翻译版本）。
2. **挖掘模型：** 采用[emrecan/bert-base-turkish-cased-mean-nli-stsb-tr](https://huggingface.co/emrecan/bert-base-turkish-cased-mean-nli-stsb-tr)模型。
3. **构建方法：**
* **编码：** 通过挖掘模型将所有查询与段落转换为稠密向量。
* **检索：** 针对每个查询，使用**Faiss**（内积相似度）检索前100个相似段落。
* **过滤：** 将真实正样本（正确答案）从检索结果中移除。
* **安全阈值：** 移除相似度得分高于**0.98**的段落，以避免出现“假负样本”（即本应为正确答案却被标记为负样本的情况）。
* **选择：** 从剩余候选段落中选取得分最高的前**10**个作为难负样本。

---

## 数据结构

### 数据示例
每一行在`.jsonl`文件中均为代表训练样本的合法JSON对象。

json
{
"query": "Manhattan projesinin başarısının hemen etkisi neydi?",
"pos": "Manhattan Projesi ve atom bombası, İkinci Dünya Savaşı'nın sona ermesine yardımcı oldu...",
"negatives": [
"Manhattan Projesi, II. Dünya Savaşı sırasında ilk atom bombasını geliştirmek için...",
"Proje, nükleer silah üretimi üzerine odaklanmıştı...",
"..."
],
"scores": [
0.874,
0.852,
"..."
]
}

### 数据字段

* **`query`**（字符串类型）：搜索查询文本。
* **`pos`**（字符串类型）：真实正样本段落（正确答案）。
* **`negatives`**（字符串列表）：包含10个与查询语义相近但并非正确答案的段落，按相似度从高到低排序。
* **`scores`**（浮点数列表）：与`negatives`列表中段落对应的余弦相似度得分，可用于训练过程中的基于间隔的过滤或加权损失函数。

-----

## 使用指南

### Python 数据集加载方法
可通过Hugging Face的`datasets`库或标准JSON行读取方式加载本数据集。

python
from datasets import load_dataset

# 若数据集已上传至Hugging Face Hub
ds = load_dataset("lumees/msmarco-tr-hard-negatives", split="train")

# 若从本地文件加载
ds = load_dataset("json", data_files="msmarco_tr_hard_negatives_final.jsonl", split="train")

print(ds[0])

### 使用Sentence Transformers进行训练
本数据集针对`MultipleNegativesRankingLoss`或`InfoNCE`等损失函数做了优化。

python
from sentence_transformers import InputExample

train_examples = []
for row in ds:
# 数据格式：[查询文本, 正样本段落, 负样本1, 负样本2, ...]
texts = [row['query'], row['pos']] + row['negatives']
train_examples.append(InputExample(texts=texts))

# 注：请确认所用损失函数支持单样本多负样本的训练场景。

-----

## 局限性与偏差

1. **翻译误差：** 原始`parsak/msmarco-tr`数据集由英语机器翻译生成，因此部分土耳其语表达可能不够自然，或存在翻译错误。
2. **假负样本风险：** 尽管设置了0.98的相似度过滤阈值，但仍有可能部分被选为“负样本”的段落实际上是原始数据集中未被标注的正确答案。
3. **模型偏差：** 负样本通过`emrecan/bert-base-turkish`模型挖掘生成，因此本数据集天然带有该基础模型的偏差与语义理解局限。

## 引用规范

若使用本数据集，请按如下格式引用Lumees AI、原始MS MARCO作者及土耳其语翻译源：

bibtex
@misc{lumees_msmarco_hn_2025,
author = {Lumees AI and Kurşun, Hasan and Yanık, Kerem Berkay},
title = {MS MARCO TR - Hard Negatives Dataset},
year = {2025},
howpublished = {url{https://lumees.io}},
}

@article{bajaj2016ms,
title={MS MARCO: A Human Generated Machine Reading Comprehension Dataset},
author={Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and others},
journal={arXiv preprint arXiv:1611.09268},
year={2016}
}

@misc{parsak_msmarco_tr,
author = {Parsak},
title = {MS MARCO Turkish Translation},
year = {2023},
publisher = {Hugging Face},
journal = {Hugging Face Hub},
howpublished = {url{https://huggingface.co/datasets/parsak/msmarco-tr}}
}