malaysia-ai/mosaic-madlad-400-ms

---
language:
- ms
---

# Mosaic format for extra dataset to train Malaysian LLM

This repository is to store dataset shards using mosaic format.

1. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-madlad-400-ms.ipynb
2. using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
3. 4096 context length.

## how-to

1. git clone,

```bash
git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-madlad-400-ms
```

2. load it,

```python
from streaming import LocalDataset
import numpy as np
from streaming.base.format.mds.encodings import Encoding, _encodings

class UInt16(Encoding):
def encode(self, obj) -> bytes:
return obj.tobytes()

def decode(self, data: bytes):
return np.frombuffer(data, np.uint16)

_encodings['uint16'] = UInt16

dataset = LocalDataset('mosaic-madlad-400-ms')
len(dataset)
```

原始信息汇总

马来西亚LLM训练的额外数据集的马赛克格式

数据集描述

• 语言: 马来语 (ms)
• 格式: 马赛克格式
• 上下文长度: 4096

数据集准备

• 准备脚本: combine-madlad-400-ms.ipynb
• 使用的分词器: bpe-tokenizer

数据集使用方法

1. 克隆数据集: bash git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-madlad-400-ms

2. 加载数据集: python from streaming import LocalDataset import numpy as np from streaming.base.format.mds.encodings import Encoding, _encodings

   class UInt16(Encoding): def encode(self, obj) -> bytes: return obj.tobytes()

    def decode(self, data: bytes):
        return np.frombuffer(data, np.uint16)
   

   _encodings[uint16] = UInt16

   dataset = LocalDataset(mosaic-madlad-400-ms) len(dataset)