dkoterwa/kor_nli_simcse

---
dataset_info:
features:
- name: premise
dtype: string
- name: entailment
dtype: string
- name: contradiction
dtype: string
splits:
- name: train
num_bytes: 90132700
num_examples: 413837
- name: valid
num_bytes: 10572025
num_examples: 48686
- name: test
num_bytes: 5289636
num_examples: 24345
download_size: 64195317
dataset_size: 105994361
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: valid
path: data/valid-*
- split: test
path: data/test-*
---
# Korean Natural Language Inference (KorNLI) for SimCSE Dataset

For a better dataset description, please visit this GitHub repository prepared by the authors of the article: [LINK](https://github.com/kakaobrain/kor-nlu-datasets) <br>
<br>
**This dataset was prepared by converting KorNLI dataset**. I took every unique premise of the dataset and searched for its entailment (positive example) and contradiction (negative example).
These changes have been made in order to apply SimCSE method.

**I additionaly share the code, which I used to convert the KorNLI dataset to make everything more clear**
```
from datasets import load_dataset
from tqdm import tqdm
import pandas as pd
import numpy as np

def create_trios(df, save_path):
list_of_examples = []
unique_premises = df.drop_duplicates("premise")["premise"]
for premise in tqdm(unique_premises):
premise_dataset = df[(df["premise"] == premise)]
positive_examples = premise_dataset[premise_dataset["label"] == "entailment"]["hypothesis"]
negative_examples = premise_dataset[premise_dataset["label"] == "contradiction"]["hypothesis"]

if len(positive_examples) == 0 or len(negative_examples) == 0:
continue

for positive_example in positive_examples:
for negative_example in negative_examples:
list_of_examples.append((premise, positive_example, negative_example))

examples_df = pd.DataFrame(list_of_examples, columns=["premise", "entailment", "contradiction"])
examples_df.to_csv(save_path)

if __name__ == "__main__":
dataset1 = load_dataset("kor_nli", "snli")["train"]
dataset2 = load_dataset("kor_nli", "multi_nli")["train"]
df_1 = pd.DataFrame(dataset1)
df_2 = pd.DataFrame(dataset2)
df_full = pd.concat([df_1, df_2])
df_full.dropna(inplace=True)
df_full["label"] = ["neutral" if label == 1 else "contradiction" if label == 2 else "entailment" for label in df_full["label"]]
create_trios(df_full, <insert your path>)
```

**How to download**

```
from datasets import load_dataset
data = load_dataset("dkoterwa/kor_nli_simcse")
```

**If you use this dataset for research, please cite this paper:**

```
@article{ham2020kornli,
title={KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding},
author={Ham, Jiyeon and Choe, Yo Joong and Park, Kyubyong and Choi, Ilji and Soh, Hyungjoon},
journal={arXiv preprint arXiv:2004.03289},
year={2020}
}
```
[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

原始信息汇总

数据集概述

数据集特征

• premise: 数据类型为字符串。
• entailment: 数据类型为字符串。
• contradiction: 数据类型为字符串。

数据集划分

• train: 包含413837个样本，总大小为90132700字节。
• valid: 包含48686个样本，总大小为10572025字节。
• test: 包含24345个样本，总大小为5289636字节。

数据集大小

• 下载大小: 64195317字节。
• 数据集总大小: 105994361字节。

配置文件

• config_name: default
  • data_files:
    • split: train, path: data/train-*
    • split: valid, path: data/valid-*
    • split: test, path: data/test-*

数据集来源

• 该数据集是通过转换KorNLI数据集创建的，用于应用SimCSE方法。