hpprc/emb

该README文件提供了多个数据集的详细信息，主要集中在它们的配置、特征和大小上。提到的数据集包括MMARCO、Mr. TyDi、MIRACL、JaGovFaqs、Auto Wiki QA、JCommonsenseQA、JSQuAD等，每个数据集都有特定的配置和特征。这些数据集用于任务如问答（QA）和检索。README还包括论文链接和联系信息以供进一步了解。

The README file provides detailed information about various datasets, primarily focusing on their configurations, features, and sizes. It mentions multiple datasets such as MMARCO, Mr. TyDi, MIRACL, JaGovFaqs, Auto Wiki QA, JCommonsenseQA, JSQuAD, and others, each with specific configurations and features. The datasets are used for tasks like QA (Question Answering) and Retrieval. The README also includes links to papers and contact information for further details.

原始信息汇总

数据集概述

1. auto-wiki-qa-collection

• 特征:
  • passage_id: int64
  • text: string
• 分割:
  • train:
    • num_bytes: 4875708765
    • num_examples: 7525391
• 下载大小: 2830807419
• 数据集大小: 4875708765

2. auto-wiki-qa-dataset

• 特征:
  • passage_id: int64
  • anc: string
  • pos_1st: string
  • neg_1st.original: null
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: null
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 20605968855
    • num_examples: 2377503
• 下载大小: 14667059765
• 数据集大小: 20605968855

3. jagovfaqs-collection

• 特征:
  • text: string
• 分割:
  • train:
    • num_bytes: 13918890
    • num_examples: 22794
• 下载大小: 5874592
• 数据集大小: 13918890

4. jagovfaqs-dataset

• 特征:
  • anc: string
  • pos_1st: string
  • neg_1st.original: null
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: null
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 205284001
    • num_examples: 22794
• 下载大小: 93115345
• 数据集大小: 205284001

5. jsquad-collection

• 特征:
  • passage_id: int64
  • text: string
• 分割:
  • train:
    • num_bytes: 3682323958
    • num_examples: 5163534
• 下载大小: 2158367417
• 数据集大小: 3682323958

6. jsquad-dataset

• 特征:
  • passage_id: int64
  • anc: string
  • pos_1st: string
  • neg_1st.original: null
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: null
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 549559659
    • num_examples: 62859
• 下载大小: 309855079
• 数据集大小: 549559659

7. miracl-collection

• 特征:
  • text: string
• 分割:
  • train:
    • num_bytes: 2699914590
    • num_examples: 6953614
• 下载大小: 1604341928
• 数据集大小: 2699914590

8. miracl-dataset

• 特征:
  • anc: string
  • pos_1st: string
  • neg_1st.original: string
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: int64
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 31809259
    • num_examples: 3477
• 下载大小: 23052176
• 数据集大小: 31809259

9. mmarco-collection

• 特征:
  • text: string
• 分割:
  • train:
    • num_bytes: 3818456967
    • num_examples: 8841823
• 下载大小: 1864051764
• 数据集大小: 3818456967

10. mmarco-dataset

• 特征:
  • anc: string
  • pos_1st: string
  • neg_1st.original: string
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: int64
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 3548802014
    • num_examples: 391060
• 下载大小: 2581211936
• 数据集大小: 3548802014

11. mr-tydi-collection

• 特征:
  • text: string
• 分割:
  • train:
    • num_bytes: 2733594839
    • num_examples: 7000027
• 下载大小: 1623335733
• 数据集大小: 2733594839

12. mr-tydi-dataset

• 特征:
  • anc: string
  • pos_1st: string
  • neg_1st.original: string
  • neg_1st.me5-large: string
  • sim_1st.me5-large: float64
  • neg_1st.bm25: string
  • sim_1st.bm25: float64
  • pos_ids: sequence: int64
  • neg_ids.original: sequence: int64
  • neg_ids.me5-large: sequence: int64
  • neg_sims.me5-large: sequence: float64
  • neg_ids.bm25: sequence: int64
  • neg_sims.bm25: sequence: float64
• 分割:
  • train:
    • num_bytes: 34930045
    • num_examples: 3697
• 下载大小: 25217483
• 数据集大小: 34930045

13. wiki-qa-collection

• 特征:
  • text: string
• 分割:
  • train:
    • num_bytes: 1182502658
    • num_examples: 2377503
• 下载大小: 669189199
• 数据集大小: 1182502658

14. wiki-qa-dataset

• 特征:
  • anc: string
  • pos_1st: string
  • neg_1st: string
  • sim_1st: float64
  • pos_ids: sequence: int64
  • neg_ids: sequence: int64
  • neg_sims: sequence: float64
• 分割:
  • train:
    • num_bytes: 10264425492
    • num_examples: 2377503
• 下载大小: 7491826894
• 数据集大小: 10264425492