huseinzol05/mosaic-yi

---
language:
- ms
---

# Mosaic format for filtered combine dataset to finetune Yi models

This repository is to store dataset shards using mosaic format.

1. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/yi/combine-dataset.ipynb
2. using tokenizer https://huggingface.co/01-ai/Yi-6B
3. 4096 context length.

## how-to

1. git clone,

```bash
git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-yi
```

2. load it,

```python
from streaming import LocalDataset
import numpy as np
from streaming.base.format.mds.encodings import Encoding, _encodings

class UInt16(Encoding):
def encode(self, obj) -> bytes:
return obj.tobytes()

def decode(self, data: bytes):
return np.frombuffer(data, np.uint16)

_encodings['uint16'] = UInt16

dataset = LocalDataset('mosaic-yi')
len(dataset)
```

原始信息汇总

数据集概述

数据集名称

Mosaic format for filtered combine dataset to finetune Yi models

数据集描述

该数据集用于存储使用mosaic格式的数据分片，旨在用于微调Yi模型。

数据集准备

• 数据集准备脚本位于：combine-dataset.ipynb
• 使用01-ai/Yi-6B的tokenizer
• 上下文长度为4096

数据集使用方法

1. 克隆数据集仓库： bash git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-yi

2. 加载数据集： python from streaming import LocalDataset import numpy as np from streaming.base.format.mds.encodings import Encoding, _encodings

   class UInt16(Encoding): def encode(self, obj) -> bytes: return obj.tobytes()

   def decode(self, data: bytes):
       return np.frombuffer(data, np.uint16)
   

   _encodings[uint16] = UInt16

   dataset = LocalDataset(mosaic-yi) len(dataset)