werty1248/multilingual-instruct-balanced

该数据集是HuggingFace Hub收集的英语、韩语、中文和日语数据集的集合，并转换为统一格式。数据可能是原生或合成的，部分数据版权不明确或仅允许非商业使用。预处理步骤包括移除过短或过长的回答以及重复的合成数据。数据格式为对话形式，包含角色和内容的字段。数据集的训练token数量使用了Mistral-Nemo-12B-Instruct tokenizer进行计算。

This dataset is a multilingual collection of English, Korean, Chinese, and Japanese datasets gathered by the HuggingFace Hub and transformed into a unified format. The data can be either native or synthetic. The dataset has undergone preprocessing to remove data with too few answer tokens or more than 8192 tokens, and synthetic data with repetitions has been removed. To balance the training tokens per language/task, the dataset has been randomly sampled. The dataset format includes conversation content and roles, suitable for text generation tasks.