tascib/turkish-instruction

--- license: cc-by-sa-4.0 language: - tr tags: - instruction-tuning - turkish - nlp - llm size_categories: - 100K<n<1M --- # Turkish Instruction Dataset ## Dataset Description This dataset is a large-scale Turkish instruction-tuning corpus created by combining multiple publicly available datasets and applying cleaning and deduplication steps. It is designed for training and evaluating large language models (LLMs) in Turkish. The dataset was prepared as part of a capstone project by students from Sabancı University. ## Data Sources The dataset is constructed from the following sources: - **TFLai / Turkish-Alpaca** (Apache-2.0) https://huggingface.co/datasets/TFLai/Turkish-Alpaca - **OpenAssistant / oasst1** (Apache-2.0) https://huggingface.co/datasets/OpenAssistant/oasst1 - **atasoglu / databricks-dolly-15k-tr** (CC BY-SA 3.0) https://huggingface.co/datasets/atasoglu/databricks-dolly-15k-tr - **merve / turkish_instructions** (Apache-2.0) https://huggingface.co/datasets/merve/turkish_instructions - **umarigan / openhermes_tr** (license not explicitly specified, derived from OpenHermes) https://huggingface.co/datasets/umarigan/openhermes_tr/tree/main ## Dataset Statistics - **Total collected samples:** 360,351 - **After deduplication:** 324,080 - **Duplicates removed:** 36,271 ### Source Breakdown - alpaca-turkish: 51,862 - oasst1-turkish: 9 - dolly-turkish: 14,714 - openhermes-turkish: 225,867 - merve-turkish: 31,628 ## Preprocessing - Merging multiple datasets - Cleaning inconsistent or malformed samples - Removing duplicates ## Intended Use This dataset is intended primarily for: - Supervised fine-tuning (SFT) of instruction-following language models - Training Turkish instruction-tuned LLMs - NLP research and experimentation in Turkish This dataset is not intended for pretraining from scratch, as it consists of instruction-response pairs rather than raw text corpora. ## Data Format Each sample follows an instruction-tuning format: - instruction: Task description given to the model - input: Optional additional context - output: Expected response This structure is compatible with common instruction-tuning pipelines such as Alpaca, Dolly, and OpenAI-style SFT datasets. ## License This dataset is released under the **CC BY-SA 4.0** license. Note: - Some source datasets are licensed under Apache-2.0 and CC BY-SA 3.0. - Derived datasets must comply with the most restrictive applicable terms. - Users must provide proper attribution and share derivative works under the same license. ## Limitations - May contain noise due to automatic aggregation - Potential biases inherited from source datasets - Uneven distribution across sources (OpenHermes dominates) ## Acknowledgements We thank the creators of the original datasets: - TFLai - OpenAssistant - Databricks / Dolly - Merve Noyan - OpenHermes contributors ## Disclaimer No responsibility or liability is accepted for the use of this dataset.--- 许可证：CC BY-SA 4.0 语言： - 土耳其语（tr） 标签： - 指令微调（instruction-tuning） - 土耳其语 - 自然语言处理（Natural Language Processing） - 大语言模型（Large Language Model） 规模类别： - 10万<样本数<100万 --- ## 土耳其语指令数据集 ## 数据集描述 本数据集为大规模土耳其语指令微调语料库，通过整合多个公开可用数据集并经过数据清洗与去重步骤构建而成，专为土耳其语大语言模型（Large Language Model）的训练与评估设计。本数据集由萨班哲大学（Sabancı University）学生的毕业综合项目所制备。 ## 数据来源 本数据集由以下数据源构建： - **TFLai / Turkish-Alpaca**（许可证：Apache-2.0） 链接：https://huggingface.co/datasets/TFLai/Turkish-Alpaca - **OpenAssistant / oasst1**（许可证：Apache-2.0） 链接：https://huggingface.co/datasets/OpenAssistant/oasst1 - **atasoglu / databricks-dolly-15k-tr**（许可证：CC BY-SA 3.0） 链接：https://huggingface.co/datasets/atasoglu/databricks-dolly-15k-tr - **merve / turkish_instructions**（许可证：Apache-2.0） 链接：https://huggingface.co/datasets/merve/turkish_instructions - **umarigan / openhermes_tr**（未明确标注许可证，源自OpenHermes） 链接：https://huggingface.co/datasets/umarigan/openhermes_tr/tree/main ## 数据集统计信息 - **总采集样本数**：360,351 - **去重后样本数**：324,080 - **移除重复样本数**：36,271 ### 数据源分布 - 土耳其语Alpaca数据集：51,862条 - 土耳其语oasst1数据集：9条 - 土耳其语Dolly数据集：14,714条 - 土耳其语OpenHermes数据集：225,867条 - 土耳其语merve指令数据集：31,628条 ## 预处理流程 - 多数据集整合 - 清洗不一致或格式错误的样本 - 移除重复样本 ## 预期用途 本数据集主要用于： - 指令跟随语言模型的监督微调（Supervised Fine-Tuning，简称SFT） - 土耳其语指令微调大语言模型的训练 - 土耳其语自然语言处理相关研究与实验 本数据集不适用于从零开始的预训练，因其仅包含指令-回复对而非原始文本语料库。 ## 数据格式 每条样本遵循指令微调格式： - `instruction`：提供给模型的任务描述 - `input`：可选的额外上下文信息 - `output`：预期生成的回复 该结构兼容常见的指令微调流程，例如Alpaca、Dolly以及OpenAI风格的监督微调数据集。 ## 许可证 本数据集采用**CC BY-SA 4.0**许可证发布。 注意事项： - 部分源数据集采用Apache-2.0与CC BY-SA 3.0许可证。 - 衍生数据集需遵守最严格的适用条款。 - 用户需提供恰当的署名，并以相同许可证协议共享衍生作品。 ## 局限性 - 因自动聚合可能包含噪声 - 可能继承源数据集的潜在偏见 - 各数据源样本分布不均（OpenHermes数据集占比极高） ## 致谢 感谢原始数据集的创作者： - TFLai - OpenAssistant - Databricks / Dolly - Merve Noyan - OpenHermes贡献者 ## 免责声明 本数据集的使用不承担任何责任或义务。