MBZUAI-Paris/DODa-10K

--- language: - ma - ar - en - fr task_categories: - translation - text2text-generation multilinguality: - multilingual language_creators: - expert-generated annotations_creators: - expert-generated - machine-generated source_datasets: - original size_categories: - 1K<n<10K license: - cc-by-nc-4.0 --- # Dataset Card for DODa-10K ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks](#supported-tasks) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) ## Dataset Description - **Homepage:** [https://hf.co/datasets/MBZUAI-Paris/DoDa-10K](https://hf.co/datasets/MBZUAI-Paris/DoDa-10K) - **Paper:** [More Information Needed] ### Dataset Summary DODa-10K is part of the Darija Open Dataset (DODa), a collaborative open-source project to collect Darija language resources. It includes translation quintuples between Darija (in both Arabic and Latin scripts), Modern Standard Arabic (MSA), English, and French. The dataset was augmented using GPT-4 for generating translations and was reviewed by native Darija speakers to ensure quality. The dataset also supports transliteration tasks for converting between Darija Arabic script and Latin script. ### Supported Tasks - Translating between Darija (Arabic and Latin), MSA, English, and French. - Transliteration between Darija Arabic script and Latin script. ### Languages The dataset includes Moroccan Arabic (Darija in both Arabic and Latin scripts), Modern Standard Arabic (MSA), English, and French. ## Dataset Structure The dataset contains translation quintuples, with each instance including the same sentence in Darija (Arabic script), Darija (Latin script), English, French, and MSA. ### Data Instances Each data instance includes: - **Darija_arabic**: The sentence in Darija, written in Arabic script. - **Darija_latin**: The same sentence in Darija, written in Latin script. - **English**: The English translation. - **French**: The French translation. - **MSA**: The Modern Standard Arabic translation. - **index**: A unique identifier for the instance. Example: ``` { "Darija_arabic": "تانقتارح أنّاك تسمع كتر أُ تهدر قلّ", "Darija_latin": "tan9tar7 annak tsm3 ktr o thdr 9ell", "English": "I suggest you listen more and talk less", "French": "Je te suggère d'écouter plus et de parler moins", "MSA": "أقترح عليك أن تستمع أكثر وتتكلم أقل", "index": 30 } ``` ## Dataset Creation ### Curation Rationale DODa-10K was created to provide a high-quality multilingual parallel corpus for Darija, MSA, English, and French, as well as to support transliteration tasks between Darija in Arabic and Latin scripts. The goal is to foster the development of multilingual models and transliteration systems for Moroccan Darija. ### Source Data #### Initial Data Collection and Normalization The dataset was collected as part of the Darija Open Dataset project, and 10,000 examples were augmented using GPT-4. The translations were then manually reviewed by native Darija speakers to ensure accuracy and quality. #### Who are the source language producers? The original DoDa dataset was augmented by GPT-4 and reviewed by native Darija speakers for accuracy. The original corpus was sourced from various open-source initiatives, such as Darija-English parallel corpora. ### Personal and Sensitive Information The dataset does not contain any personal or sensitive information. ## Considerations for Using the Data ### Social Impact of Dataset This dataset promotes the development of NLP tools for Moroccan Darija, a language with limited digital resources. It also facilitates research on translation and transliteration between several languages to Moroccan Darija. ### Discussion of Biases The dataset was machine-generated and manually reviewed, but it may still reflect biases present in the machine translation models used, particularly in cultural nuances between the languages. ### Other Known Limitations - The dataset is limited to the first 10K examples from the Darija Open Dataset project. ## Additional Information ### Dataset Curators - MBZUAI-Paris team ### Licensing Information - [Creative Commons Attribution-NonCommercial 4.0 International License](https://github.com/Gibberlings3/GitHub-Templates/blob/master/License-Templates/CC-BY-NC-ND-4.0/LICENSE-CC-BY-NC-ND-4.0.md) ### Citation Information ``` @article{shang2024atlaschatadaptinglargelanguage, title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect}, author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing}, year={2024}, eprint={2409.17912}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2409.17912}, } ``` ``` @article{outchakoucht2024evolution, title={The Evolution of Darija Open Dataset: Introducing Version 2}, author={Outchakoucht, Aissam and Es-Samaali, Hamza}, journal={arXiv preprint arXiv:2405.13016}, year={2024} } ``` ``` @article{outchakoucht2021moroccan, title={Moroccan dialect-darija-open dataset}, author={Outchakoucht, Aissam and Es-Samaali, Hamza}, journal={arXiv preprint arXiv:2103.09687}, year={2021} } ```--- 语言: - 摩洛哥阿拉伯语（ma） - 阿拉伯语（ar） - 英语（en） - 法语（fr） 任务类别: - 翻译 - 文本到文本生成 多语言类型: - 多语言 语言生成者: - 专家生成 标注生成者: - 专家生成 - 机器生成 源数据集: - 原始数据集 数据规模分类: - 1000 < 样本数 < 10000 许可证: - CC-BY-NC-4.0 --- # DODa-10K 数据集卡片 ## 目录 - [目录](#目录) - [数据集描述](#数据集描述) - [数据集概述](#数据集概述) - [支持任务](#支持任务) - [语言](#语言) - [数据集结构](#数据集结构) - [数据样本](#数据样本) - [数据集构建](#数据集构建) - [构建依据](#构建依据) - [源数据](#源数据) - [个人与敏感信息](#个人与敏感信息) - [数据集使用注意事项](#数据集使用注意事项) - [数据集的社会影响](#数据集的社会影响) - [偏差讨论](#偏差讨论) - [其他已知局限性](#其他已知局限性) - [附加信息](#附加信息) - [数据集构建者](#数据集构建者) - [许可证信息](#许可证信息) - [引用信息](#引用信息) ## 数据集描述 - **主页**: [https://hf.co/datasets/MBZUAI-Paris/DoDa-10K](https://hf.co/datasets/MBZUAI-Paris/DoDa-10K) - **论文**: [更多信息待补充] ### 数据集概述 DODa-10K 属于达里贾语开放数据集（Darija Open Dataset, DODa），是一个用于收集达里贾语语言资源的协作式开源项目。该数据集包含达里贾语（阿拉伯字母与拉丁字母两种书写形式）、现代标准阿拉伯语（Modern Standard Arabic, MSA）、英语与法语之间的翻译五元组。数据集通过GPT-4生成翻译内容进行扩充，并由达里贾语母语使用者审核以确保质量。此外，该数据集还支持达里贾语阿拉伯字母与拉丁字母之间的转写任务。 ### 支持任务 - 达里贾语（阿拉伯字母与拉丁字母形式）、MSA、英语与法语之间的互译； - 达里贾语阿拉伯字母与拉丁字母之间的转写。 ### 语言 本数据集包含摩洛哥阿拉伯语（达里贾语，支持阿拉伯字母与拉丁字母两种书写形式）、现代标准阿拉伯语（MSA）、英语以及法语。 ## 数据集结构 本数据集包含翻译五元组，每个样本均包含同一语句的五种形式：达里贾语（阿拉伯字母书写）、达里贾语（拉丁字母书写）、英语译文、法语译文以及MSA译文。 ### 数据样本 每个数据样本包含以下字段： - **Darija_arabic**: 以阿拉伯字母书写的达里贾语语句； - **Darija_latin**: 以拉丁字母书写的同一达里贾语语句； - **English**: 英语译文； - **French**: 法语译文； - **MSA**: 现代标准阿拉伯语译文； - **index**: 样本的唯一标识符。 示例： { "Darija_arabic": "تانقتارح أنّاك تسمع كتر أُ تهدر قلّ", "Darija_latin": "tan9tar7 annak tsm3 ktr o thdr 9ell", "English": "I suggest you listen more and talk less", "French": "Je te suggère d'écouter plus et de parler moins", "MSA": "أقترح عليك أن تستمع أكثر وتتكلم أقل", "index": 30 } ## 数据集构建 ### 构建依据 构建DODa-10K的目的是为达里贾语、MSA、英语与法语提供高质量的多语言平行语料库，同时支持达里贾语阿拉伯字母与拉丁字母之间的转写任务。其目标是推动面向摩洛哥达里贾语的多语言模型与转写系统的研发。 ### 源数据 #### 初始数据收集与标准化 本数据集作为达里贾语开放数据集项目的一部分进行收集，并通过GPT-4扩充至10000条样本。随后由达里贾语母语使用者对翻译内容进行人工审核，以确保准确性与质量。 #### 源语言生产者是谁？ 原始DoDa数据集由GPT-4进行扩充，并由达里贾语母语使用者审核以保证准确性。原始语料库来源于多个开源项目，例如达里贾语-英语平行语料库。 ### 个人与敏感信息 本数据集未包含任何个人或敏感信息。 ## 数据集使用注意事项 ### 数据集的社会影响 本数据集有助于推动针对摩洛哥达里贾语的自然语言处理工具研发——该语言的数字化资源相对匮乏。同时，它也促进了针对多语言与摩洛哥达里贾语之间的翻译与转写相关研究。 ### 偏差讨论 本数据集由机器生成并经人工审核，但仍可能反映所使用的机器翻译模型中存在的偏差，尤其是在不同语言间的文化细微差异方面。 ### 其他已知局限性 - 本数据集仅包含达里贾语开放数据集项目中的前10000条样本。 ## 附加信息 ### 数据集构建者 - MBZUAI-Paris团队 ### 许可证信息 - [知识共享署名-非商业性使用4.0国际许可协议（Creative Commons Attribution-NonCommercial 4.0 International License）](https://github.com/Gibberlings3/GitHub-Templates/blob/master/License-Templates/CC-BY-NC-ND-4.0/LICENSE-CC-BY-NC-ND-4.0.md) ### 引用信息 @article{shang2024atlaschatadaptinglargelanguage, title={Atlas-Chat: 适配低资源摩洛哥阿拉伯语方言的大语言模型}, author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing}, year={2024}, eprint={2409.17912}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2409.17912}, } @article{outchakoucht2024evolution, title={达里贾语开放数据集的演进：版本2发布}, author={Outchakoucht, Aissam and Es-Samaali, Hamza}, journal={arXiv预印本 arXiv:2405.13016}, year={2024} } @article{outchakoucht2021moroccan, title={摩洛哥方言——达里贾语开放数据集}, author={Outchakoucht, Aissam and Es-Samaali, Hamza}, journal={arXiv预印本 arXiv:2103.09687}, year={2021} }