malaysia-ai/mosaic-extra

---
language:
- ms
---

# Mosaic format for extra dataset to train Malaysian LLM

This repository is to store dataset shards using mosaic format.

1. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-extra.ipynb
2. using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
3. 4096 context length.

## how-to

1. git clone,

```bash
git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-extra
```

2. load it,

```python
from streaming import LocalDataset
import numpy as np
from streaming.base.format.mds.encodings import Encoding, _encodings

class UInt16(Encoding):
def encode(self, obj) -> bytes:
return obj.tobytes()

def decode(self, data: bytes):
return np.frombuffer(data, np.uint16)

_encodings['uint16'] = UInt16

dataset = LocalDataset('mosaic-extra')
len(dataset)
```

原始信息汇总

数据集概述

数据集描述

该数据集用于训练马来西亚的大型语言模型（LLM），采用马赛克格式存储数据分片。

数据集准备

• 数据集准备脚本位于：combine-extra.ipynb
• 使用分词器：bpe-tokenizer
• 上下文长度为4096。

数据集使用方法

1. 克隆数据集： bash git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-extra

2. 加载数据集： python from streaming import LocalDataset import numpy as np from streaming.base.format.mds.encodings import Encoding, _encodings

   class UInt16(Encoding): def encode(self, obj) -> bytes: return obj.tobytes()

    def decode(self, data: bytes):
        return np.frombuffer(data, np.uint16)
   

   _encodings[uint16] = UInt16

   dataset = LocalDataset(mosaic-extra) len(dataset)