hotchpotch/msmarco-ja-hard-negatives

该数据集是基于MS MARCO的日语翻译数据，经过硬负样本挖掘处理，并用于信息检索模型SPLADE的训练。数据集包含两个配置：collection和dataset。collection配置包含文本和其英文翻译，dataset配置包含锚点文本、其英文翻译、正例ID、负例ID及其相似度等信息。数据集经过Unicode标准化、过滤和选择处理，并使用japanese-splade-base-v1-mmarco-only模型进行硬负样本挖掘。数据集还通过bge-reranker-v2-m3模型进行评分，并与mMARCO数据集进行了比较。评估结果显示，该数据集在多个指标上优于mMARCO数据集。

This dataset is the result of hard negative mining on Japanese translation data from MS MARCO. The processing includes normalization, filtering, and selection of translation data, as well as extraction and scoring of hard negatives. Evaluation results show that this dataset outperforms the mmarco(ja) dataset in multiple evaluation metrics. Notes include potential issues with the scores of hard negatives and positive samples, and it is recommended to use score thresholds for filtering.