pietrolesci/joci
收藏资源简介：
该数据集是完整的JOCI数据集，文件名为`joci.csv.zip`。数据集经过了特定的处理，包括将`label`列重命名为`original_label`，并创建新的`label`列，使用映射规则将原始标签转换为常见的NLI（自然语言推理）类别，最终将标签转换为`entailment`、`neutral`和`contradiction`三类。数据集包含了上下文、假设、标签、原始标签、上下文来源、假设来源和子集等信息。
This dataset is the complete JOCI dataset, with the file name `joci.csv.zip`. It has undergone targeted preprocessing, including renaming the original `label` column to `original_label` and creating a new `label` column. A mapping rule is utilized to convert the original labels into standard Natural Language Inference (NLI) categories, ultimately categorizing the new labels into three classes: `entailment`, `neutral`, and `contradiction`. The dataset includes fields such as context, hypothesis, label, original_label, context source, hypothesis source, and subset.
数据集概述
本数据集为“full” JOCI数据集，文件名为joci.csv.zip，原始数据集可在此链接获取。
数据集处理
-
将
label列重命名为original_label。 -
创建新的
label列，使用以下映射关系进行转换：{ 0: "contradiction", 1: "contradiction",
2: "neutral", 3: "neutral", 4: "neutral", 5: "entailment", } -
将标签转换为NLI标准类别：
{"entailment": 0, "neutral": 1, "contradiction": 2}。
数据集创建代码
python import pandas as pd from datasets import Features, Value, ClassLabel, Dataset
读取数据
df = pd.read_csv("<path to folder>/joci.csv")
列名转换为小写
df.columns = df.columns.str.lower()
重命名标签列
df = df.rename(columns={"label": "original_label"})
编码标签
df["label"] = df["original_label"].map({
0: "contradiction",
1: "contradiction",
2: "neutral",
3: "neutral",
4: "neutral",
5: "entailment",
})
编码标签
df["label"] = df["label"].map({"entailment": 0, "neutral": 1, "contradiction": 2})
转换为数据集格式
features = Features({ "context": Value(dtype="string"), "hypothesis": Value(dtype="string"), "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"]), "original_label": Value(dtype="int32"), "context_from": Value(dtype="string"), "hypothesis_from": Value(dtype="string"), "subset": Value(dtype="string"), }) ds = Dataset.from_pandas(df, features=features) ds.push_to_hub("joci", token="<token>")
