NousResearch/dolma-v1_7-30B-tokenized-llama2-nanoset
收藏Hugging Face2024-05-30 更新2024-06-12 收录
下载链接：
https://hf-mirror.com/datasets/NousResearch/dolma-v1_7-30B-tokenized-llama2-nanoset
下载链接
链接失效反馈官方服务：
资源简介：
---
license: odc-by
task_categories:
- text-generation
language:
- en
tags:
- language-modeling
- casual-lm
- llm
pretty_name: Dolma
size_categories:
- 100B<n<1T
---
Tokenized (Llama 2) verison of [NousResearch/dolma-v1_7-30B](https://huggingface.co/datasets/NousResearch/dolma-v1_7-30B) as a [Nanotron](https://github.com/huggingface/nanotron) dataset split into 10 GB chunks.
To download:
```shell
huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-30B-tokenized-llama2-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-30B-tokenized-llama2-nanoset
```
To recombine:
```shell
cat dolma-v1_7-30B-tokenized-llama2-nanoset/dolma-v1_7-30B-tokenized-llama2-nanoset_input_ids.npy.* > dolma-v1_7-30B-tokenized-llama2-nanoset.npy
rm -rf dolma-v1_7-30B-tokenized-llama2-nanoset
```
Can also be used directly with numpy, for example
```python
import numpy as np
dataset_buffer_mmap = np.memmap("dolma-v1_7-30B-tokenized-llama2-nanoset.npy",
mode="r", order="C", dtype=np.int32)
dataset_buffer = memoryview(dataset_buffer_mmap)
dataset_number_of_tokens = int(len(dataset_buffer))
```
提供机构：
NousResearch
原始信息汇总
数据集概述
基本信息
- 许可证: ODC-BY
- 任务类别: 文本生成
- 语言: 英语
- 标签: 语言建模, 休闲语言模型, 大型语言模型
- 美观名称: Dolma
- 大小类别: 100B<n<1T
数据集描述
- 版本: Dolma v1.7-30B 的 Tokenized (Llama 2) 版本
- 格式: 分割成10 GB 的块，作为 Nanotron 数据集
- 来源: NousResearch/dolma-v1_7-30B
下载与重组
-
下载命令: shell huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-30B-tokenized-llama2-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-30B-tokenized-llama2-nanoset
-
重组命令: shell cat dolma-v1_7-30B-tokenized-llama2-nanoset/dolma-v1_7-30B-tokenized-llama2-nanoset_input_ids.npy.* > dolma-v1_7-30B-tokenized-llama2-nanoset.npy rm -rf dolma-v1_7-30B-tokenized-llama2-nanoset
使用示例
-
使用 numpy 直接操作: python import numpy as np
dataset_buffer_mmap = np.memmap("dolma-v1_7-30B-tokenized-llama2-nanoset.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer))
