wenbopan/RefGPT-Fact-v2-8x

--- license: apache-2.0 dataset_info: features: - name: dialogue dtype: string - name: reference dtype: string - name: language dtype: string - name: type dtype: string splits: - name: en num_bytes: 303791610 num_examples: 7322 - name: zh num_bytes: 105454150 num_examples: 7582 download_size: 232926198 dataset_size: 409245760 configs: - config_name: default data_files: - split: en path: data/en-* - split: zh path: data/zh-* --- # Dataset Card for "RefGPT-Fact-v2-8x" This is a lengthened version of [Mutonix/RefGPT-Fact-v2](https://huggingface.co/datasets/Mutonix/RefGPT-Fact-v2). The reference field in each sample is 8 times the length of the original sample. Correspondingly, the dataset is subsampled to 1/8 of its original size. ## Data Construction Each reference is upsampled by K times, where: $$ K \sim \text{Poisson}(\lambda=8) $$ To lengthen the reference text, the reference text is shuffled into K - 1 paragraphs. The K - 1 paragraphs are the nearest samples to the target text, where the distances are calculated using OpenAI's `text-embedding-3-small` and FAISS.

原始信息汇总

数据集概述

数据集名称

RefGPT-Fact-v2-8x

数据集信息

• 许可证: Apache-2.0
• 特征:
  • dialogue: 数据类型为 string
  • reference: 数据类型为 string
  • language: 数据类型为 string
  • type: 数据类型为 string
• 分割:
  • en: 字节数为 303791610，示例数为 7322
  • zh: 字节数为 105454150，示例数为 7582
• 下载大小: 232926198
• 数据集大小: 409245760

配置

• 默认配置:
  • en: 数据文件路径为 data/en-*
  • zh: 数据文件路径为 data/zh-*

数据构造

• 参考字段通过泊松分布（λ=8）进行上采样，每个参考文本被扩展为K-1个段落，其中K-1个段落是从与目标文本距离最近的样本中选取，距离计算使用OpenAI的text-embedding-3-small和FAISS算法。