NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset

---
license: odc-by
task_categories:
- text-generation
language:
- en
tags:
- language-modeling
- casual-lm
- llm
pretty_name: Dolma
size_categories:
- 100B<n<1T
---

Tokenized (Llama 3) verison of [NousResearch/dolma-v1_7-305B](https://huggingface.co/datasets/NousResearch/dolma-v1_7-305B) as a [Nanotron](https://github.com/huggingface/nanotron) dataset split into 10 GB chunks.

To download:
```shell
huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset
```

To recombine:
```shell
cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy
rm -rf dolma-v1_7-305B-tokenized-llama3-nanoset
```

Can also be used directly with numpy, for example
```python
import numpy as np

dataset_buffer_mmap = np.memmap("dolma-v1_7-305B-tokenized-llama3-nanoset.npy",
mode="r", order="C", dtype=np.int32)
dataset_buffer = memoryview(dataset_buffer_mmap)
dataset_number_of_tokens = int(len(dataset_buffer))
```

原始信息汇总

数据集概述

基本信息

• 许可证: odc-by
• 任务类别: text-generation
• 语言: en
• 标签:
  • language-modeling
  • casual-lm
  • llm
• 名称: Dolma
• 大小类别: 100B<n<1T

数据集描述

• 版本: Dolma-v1_7-305B-tokenized-llama3-nanoset
• 格式: 分为10 GB的块，使用Nanotron格式
• 原始数据集: NousResearch/dolma-v1_7-305B
• 处理方式: 使用Llama 3进行Token化处理

使用方法

• 下载命令: shell huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset

• 重组命令: shell cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy rm -rf dolma-v1_7-305B-tokenized-llama3-nanoset

• 直接使用示例: python import numpy as np

  dataset_buffer_mmap = np.memmap("dolma-v1_7-305B-tokenized-llama3-nanoset.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer))