rulins/MassiveDS-1.4T

MassiveDS数据集包含原始段落、嵌入和索引。该数据集有两个版本：MassiveDS-1.4T和MassiveDS-140B。MassiveDS-1.4T包含1.4T的标记，而MassiveDS-140B是其子采样版本，包含140B的标记。文件结构包括`raw_data`（JSONL格式的原始数据）、`passages`（分块的原始段落，每个段落不超过256个单词）、`embeddings`（使用Contriever-MSMACRO编码的段落嵌入）和`index`（基于嵌入构建的平面索引）。建议使用Git LFS下载大文件，并提供了下载示例脚本。

The MassiveDS dataset offers two versions: MassiveDS-1.4T contains 1.4 trillion tokens, and MassiveDS-140B is a subsampled version containing 140 billion tokens. The dataset includes raw data, passages, embeddings, and index. The raw data is stored in JSONL files, passages are chunked into no more than 256 words with passage IDs. Embeddings are encoded using Contriever-MSMACRO, and the index is a flat index built from these embeddings. The dataset supports downloading via Git LFS, with options for partial downloads.