nixiesearch/bfhnd-small

---
language:
- en
license: apache-2.0
tags:
- text
pretty_name: "BFHND: Big Hard Negatives Dataset (1M sample)"
size_categories:
- "100K<n<1M"
source_datasets:
- "BeIR"
task_categories:
- sentence-similarity
dataset_info:
config_name: default
features:
- name: query
dtype: string
- name: positive
sequence: string
- name: negative
sequence: string
splits:
- name: train
num_bytes: 226515502
num_examples: 1000000
train-eval-index:
- config: default
task: sentence-similarity
splits:
train_split: train
configs:
- config_name: default
data_files:
- split: train
path: "data/train/*"
---

# Big Hard Negatives Dataset

A dataset for training embedding models for semantic search.

TODO: add desc

A dataset in a [nixietune](https://github.com/nixiesearch/nixietune) compatible format:

```json
{
"query": ")what was the immediate impact of the success of the manhattan project?",
"pos": [
"The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated."
],
"neg": [
"Abstract. The pivotal engineering and scientific success of the Twentieth century was the Manhattan Project. The Manhattan Project assimilated concepts and leaders from all scientific fields and engineering disciplines to construct the first two atomic bombs.",
"The pivotal engineering and scientific success of the Twentieth century was the Manhattan Project. The Manhattan Project assimilated concepts and leaders from all scientific fields and engineering disciplines to construct the first two atomic bombs."
]
}
```

## Usage

To use with HF datasets:

```bash
pip install datasets zstandard
```

```python
from datasets import load_dataset

data = load_dataset('nixiesearch/bfhardneg-small')
print(data["train"].features)
```

## License

Apache 2.0

原始信息汇总

Big Hard Negatives Dataset

概述

• 名称: BFHND: Big Hard Negatives Dataset (1M sample)
• 语言: 英语
• 许可证: Apache 2.0
• 标签: 文本
• 大小类别: 100K<n<1M
• 来源数据集: BeIR
• 任务类别: 句子相似度

数据集信息

• 配置名称: default
• 特征:
  • query: 字符串
  • positive: 字符串序列
  • negative: 字符串序列
• 分割:
  • train:
    • 字节数: 226515502
    • 样本数: 1000000

训练与评估索引

• 配置: default
• 任务: 句子相似度
• 分割:
  • 训练分割: train

配置

• 配置名称: default
• 数据文件:
  • 分割: train
  • 路径: "data/train/*"