MBZUAI-Paris/DarijaStory

--- language: - ary task_categories: - text-generation multilinguality: - monolingual language_creators: - machine-translated source_datasets: - original size_categories: - 1K<n<10K --- # Dataset Card for DarijaStory ## Table of Contents - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) ## Dataset Description - **Repository:** [https://hf.co/datasets/MBZUAI-Paris/DarijaStory](https://hf.co/datasets/MBZUAI-Paris/DarijaStory) - **Paper:** [https://arxiv.org/pdf/2409.17912](https://arxiv.org/pdf/2409.17912) ### Dataset Summary DarijaStory is a story completion dataset. It consists of 4,392 long stories scraped from [9esa](https://www.9esa.com), a website featuring a variety of stories written in Moroccan Darija. ### Supported Tasks and Leaderboards - **Task Category:** Conditional Text Generation - **Task:** Story Completion in Moroccan Darija ### Languages The dataset is available in Moroccan Arabic (Darija). ## Dataset Structure ### Data Instances Each data instance contains a story or a chapter of story. #### Example Data Instance: ``` { 'id': 1170, 'story_name': 'قصة اللؤلؤة السوداء', 'content': 'حلات عوييناتها بتقااالة... حاسة بحرييق كيقطع فراااسها... قوي وعينيها مضببين ليها رؤيا ... هزات يديها بتقالة حطاتها فوق رااسها....' } ``` ### Data Fields - **id**: *(integer)* Index of the story. - **story_name**: *(string)* The story name. - **content**: *(integer)* The content of the story. ### Data Splits The dataset consists of a single split: | Split | Number of Instances | |-------|----------------------| | train | 4,392 | ## Dataset Creation ### Curation Rationale The dataset was web-scraped from [9esa.com](https://www.9esa.com) that contains stories in Darija. ### Personal and Sensitive Information The dataset does not contain personal, private, or sensitive information. All stories are general and cover fictional or societal themes relevant to Morocco. ## Considerations for Using the Data ### Social Impact of Dataset This dataset promotes the development and evaluation of language models capable of understanding and generating extended narratives in Moroccan Darija, thus contributing to the advancement of NLP in underrepresented languages and supporting cultural diversity in AI applications. ### Discussion of Biases The dataset consists of Moroccan Darija stories, which may reflect specific cultural and societal themes relevant to Morocco. Users should be aware of this when using the dataset for general language model applications. ## Additional Information ### Dataset Curators - **MBZUAI-Paris Team** ### Licensing Information - **License:** [ODC-BY](https://opendatacommons.org/licenses/by/1-0/). ### Citation If you use this dataset in your research, please cite our paper: ```none @article{shang2024atlaschatadaptinglargelanguage, title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect}, author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing}, year={2024}, eprint={2409.17912}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2409.17912}, } ```--- 语言： - ary 任务类别： - 文本生成 多语言属性： - 单语言 语言来源： - 机器翻译 源数据集： - 原始数据集 样本规模区间： - 1K < n < 10K --- # 达里贾语故事（DarijaStory）数据集卡片 ## 目录 - [数据集描述](#dataset-description) - [数据集摘要](#dataset-summary) - [支持任务与排行榜](#supported-tasks-and-leaderboards) - [语言](#languages) - [数据集结构](#dataset-structure) - [数据实例](#data-instances) - [数据字段](#data-fields) - [数据拆分](#data-splits) - [数据集构建](#dataset-creation) - [遴选依据](#curation-rationale) - [个人与敏感信息](#personal-and-sensitive-information) - [数据使用注意事项](#considerations-for-using-the-data) - [数据集的社会影响](#social-impact-of-dataset) - [偏差讨论](#discussion-of-biases) - [附加信息](#additional-information) - [数据集构建者](#dataset-curators) - [许可信息](#licensing-information) - [引用信息](#citation-information) ## 数据集描述 - **仓库地址：** [https://hf.co/datasets/MBZUAI-Paris/DarijaStory](https://hf.co/datasets/MBZUAI-Paris/DarijaStory) - **论文链接：** [https://arxiv.org/pdf/2409.17912](https://arxiv.org/pdf/2409.17912) ### 数据集摘要 达里贾语故事（DarijaStory）是一款故事补全数据集，其包含从[9esa](https://www.9esa.com)爬取的4392篇长故事，该网站汇集了诸多以摩洛哥达里贾语（Darija）创作的各类故事。 ### 支持任务与排行榜 - **任务类别：** 条件文本生成 - **任务：** 摩洛哥达里贾语故事补全 ### 语言 本数据集采用摩洛哥阿拉伯语（达里贾语，Darija）。 ## 数据集结构 ### 数据实例 每个数据实例包含一篇故事或故事章节。 #### 数据实例示例： { 'id': 1170, 'story_name': 'قصة اللؤلؤة السوداء', 'content': 'حلات عوييناتها بتقااالة... حاسة بحرييق كيقطع فراااسها... قوي وعينيها مضببين ليها رؤيا ... هزات يديها بتقالة حطاتها فوق رااسها....' } ### 数据字段 - **id**: *(整数)* 故事的索引。 - **story_name**: *(字符串)* 故事名称。 - **content**: *(整数)* 故事内容。 ### 数据拆分 本数据集仅包含一个拆分： | 拆分名称 | 实例数量 | |-------|----------------------| | 训练集（train） | 4,392 | ## 数据集构建 ### 遴选依据 本数据集从[9esa.com](https://www.9esa.com)爬取而来，该网站包含达里贾语故事。 ### 个人与敏感信息 本数据集未包含任何个人、私密或敏感信息，所有故事均为通用内容，涵盖与摩洛哥相关的虚构或社会主题。 ## 数据使用注意事项 ### 数据集的社会影响 本数据集可推动能够理解并生成摩洛哥达里贾语长篇叙事文本的大语言模型（Large Language Model）的开发与评估，助力低资源语言自然语言处理（Natural Language Processing，NLP）领域的发展，并支撑人工智能应用中的文化多样性。 ### 偏差讨论 本数据集由摩洛哥达里贾语故事组成，可能反映与摩洛哥相关的特定文化与社会主题，用户在将该数据集用于通用大语言模型应用时应注意这一点。 ## 附加信息 ### 数据集构建者 - **MBZUAI-Paris团队** ### 许可信息 - **许可协议：** [ODC-BY](https://opendatacommons.org/licenses/by/1-0/)。 ### 引用信息 如果您在研究中使用该数据集，请引用我们的论文： none @article{shang2024atlaschatadaptinglargelanguage, title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect}, author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing}, year={2024}, eprint={2409.17912}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2409.17912}, }