multilingual-tinystories

Multilingual TinyStories 是一个包含多种印度语言儿童故事的数据集，专为语言模型训练而生成。数据集涵盖17种印度语言，包括阿萨姆语、多格里语、古吉拉特语、卡纳达语、马拉雅拉姆语、曼尼普尔语、尼泊尔语、奥里亚语、旁遮普语、梵语、桑塔利语、信德语、泰米尔语、泰卢固语和乌尔都语等。总共有132,942个故事，93,909,863个标记。每个故事以原生脚本存储，并配有唯一的索引标识符。数据集适用于小型语言模型训练、多语言语言模型研究、跨语言迁移学习和教育应用。数据通过语言模型生成，经过清理以确保内容适合儿童阅读，但可能存在文化或语言上的不准确之处。数据集采用CC-BY-4.0许可。

Multilingual TinyStories is a dataset of children's stories in multiple Indian languages, specifically created for language model training. The dataset covers 17 Indian languages, including Assamese, Dogri, Gujarati, Kannada, Malayalam, Manipuri, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, Urdu, and others. In total, it contains 132,942 stories and 93,909,863 tokens. Each story is stored in its native script and assigned a unique index identifier. The dataset is suitable for small language model training, multilingual language model research, cross-lingual transfer learning, and educational applications. The data was generated using language models and cleaned to ensure content appropriate for children, though potential cultural or linguistic inaccuracies may still exist. The dataset is licensed under CC-BY-4.0.