malaysia-ai/mosaic-dedup-text-dataset-filtered

--- language: - ms --- # Mosaic format for filtered dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. 1. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-filtered-4096.ipynb 2. using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer 3. 4096 context length. ## how-to 1. git clone, ```bash git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset ``` 2. load it, ```python from streaming import LocalDataset import numpy as np from streaming.base.format.mds.encodings import Encoding, _encodings class UInt16(Encoding): def encode(self, obj) -> bytes: return obj.tobytes() def decode(self, data: bytes): return np.frombuffer(data, np.uint16) _encodings['uint16'] = UInt16 dataset = LocalDataset('mosaic-dedup-text-dataset-filtered') len(dataset) ```

原始信息汇总

数据集概述

数据集名称

Mosaic format for filtered dedup text dataset to train Malaysian LLM

数据集描述

该数据集用于存储使用mosaic格式的数据分片，旨在训练马来西亚的大型语言模型。

数据集准备

• 数据集准备脚本位于：combine-dedup-text-dataset-filtered-4096.ipynb
• 使用tokenizer：malaysia-ai/bpe-tokenizer
• 上下文长度为4096。

数据集使用方法

1. 克隆数据集： bash git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset

2. 加载数据集： python from streaming import LocalDataset import numpy as np from streaming.base.format.mds.encodings import Encoding, _encodings

   class UInt16(Encoding): def encode(self, obj) -> bytes: return obj.tobytes()

    def decode(self, data: bytes):
        return np.frombuffer(data, np.uint16)
   

   _encodings[uint16] = UInt16

   dataset = LocalDataset(mosaic-dedup-text-dataset-filtered) len(dataset)