MBZUAI/instructpoet-ar

--- pretty_name: Arabic Poetry IFT language: - ar language_creators: - found license: other multilinguality: multilingual size_categories: - 1M<n<10M source_datasets: - original task_categories: - text-generation - question-answering task_ids: - text2text-generation - multiple-choice-qa annotations_creators: - expert-generated - machine-generated tags: - arabic - poetry - instruction-following - dialects - literary-text configs: - config_name: generation_templates default: true data_files: - split: generation_templates path: "Templates - Poetry Generation.csv" - config_name: continuation_templates data_files: - split: continuation_templates path: "Templates - Poetry Continuation.csv" - config_name: analysis_templates data_files: - split: analysis_templates path: "Templates - Poetry Analysis.csv" - config_name: corruption_templates data_files: - split: corruption_templates path: "Templates - Poetry Corruption.csv" --- # Arabic Poetry IFT ## Dataset Summary Arabic Poetry IFT is a large-scale instruction-following dataset for Arabic poetry understanding and co-creation. It supports four task families: generation, continuation, revision/restoration, and multiple-choice analysis. The dataset covers Modern Standard Arabic (MSA) and four regional Arabic varieties used in the instruction layer: Gulf, Levantine, Nile Valley, and North African Arabic. This release accompanies the ACL 2026 paper *Instruction-Guided Poetry Generation in Arabic and Its Dialects*. **Dataset curators:** Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto. ### Supported Tasks 1. `generation`: compose a poem from constraints such as title, poet, era, genre, meter, rhyme, keywords, or key phrases 2. `continuation`: continue a partial poem while preserving poetic constraints 3. `revision` / `corruption`: restore a corrupted poem to its intended form 4. `analysis`: answer multiple-choice questions about poem metadata such as poet, title, keywords, meter, era, genre, and rhyme ### Language Coverage - Modern Standard Arabic (MSA) - Gulf Arabic - Levantine Arabic - Nile Valley Arabic - North African Arabic Most source poems are in MSA, while dialectal coverage is introduced primarily through manually written instruction templates and a smaller amount of dialectal source material. ## Dataset Construction ### Source Collection The underlying poetry corpus was aggregated from public literary resources, unified into a common format, enriched with metadata, and deduplicated before instruction generation. Poems with only one verse were removed. Rhyme was automatically inferred when at least 70% of verse endings matched after normalization. Training corpus statistics from the paper: | Source | Train poems | Avg. verses | |---|---:|---:| | Ashaar | 123,581 | 19.81 | | PoetsGate* | 112,482 | 15.58 | | Adab* | 70,277 | 35.33 | | AraPoems | 62,963 | 22.01 | | Diwan* | 38,005 | 22.65 | | Mawsooaa* | 18,002 | 10.25 | | Arapoet* | 1,303 | 9.25 | | Arabic Poetry Dataset | 662 | 19.41 | | Arabic-Poetry-Melody | 48 | 21.44 | | Adab World* | 6 | 93.33 | | Other | 8 | 24.88 | | **Total** | **427,337** | **21.39** | Test benchmark statistics from the paper: | Source | Test poems | Avg. verses | |---|---:|---:| | FannOrFlop (Al Ghallabi et al., 2025) | 6,984 | 17.97 | \* Scraped sources, as reported in the paper. ### Metadata Enrichment The corpus uses both original and derived metadata. Key fields used to build the instruction tasks include: - `poem_text` - `poem_title` - `poet_name` - `poet_era` - `genre` - `meter` - `rhyme` - `keywords` - `key_phrases` `keywords` and `key_phrases` were automatically generated using Gemini 2.5 Pro. The paper reports a manual quality check on 100 sampled poems, where 96% of extracted keywords were judged to be good quality. ### De-duplication and Leakage Control - Intra-source duplicates were removed after normalization. - Orthographic normalization included removing elongation, removing diacritics, and standardizing spelling variants. - Any poem overlapping with the FannOrFlop benchmark was removed from training to avoid train/test leakage. ### Instruction Template Creation The instruction layer was manually designed for four task families and then expanded across five Arabic varieties. The camera-ready paper reports: | Task family | Base templates | Arabic varieties | Dialect-specific templates | |---|---:|---:|---:| | Generation | 246 | 5 | 1,230 | | Continuation | 176 | 5 | 880 | | Analysis | 214 | 5 | 1,070 | | Revision | 8 | 5 | 40 | | **Total** | **644** | **5** | **3,220** | Dialect templates were written and revised by native speakers of the corresponding regional varieties. ## Dataset Statistics ### Overall IFT Size The instruction-following dataset contains **1,350,897** training pairs and **24,815** test pairs, for **1,375,712** total examples. | Task | Train examples | Test examples | Train subtasks | Test subtasks | |---|---:|---:|---:|---:| | Generation | 427,337 | 6,984 | 19 | 19 | | Continuation | 427,276 | 6,984 | 11 | 11 | | Revision | 68,947 | 3,863 | 8 | 8 | | Analysis | 427,337 | 6,984 | 16 | 14 | ### Task-Specific Notes - Continuation examples are created by splitting poems at random cut points between 10% and 90% of the poem length. - Analysis is framed as multiple-choice question answering with 1 correct answer and 4 distractors. - Revision examples are generated from automatically corrupted poems paired with their clean originals. The eight revision corruption types are: - `rhyme_structure` - `full_style` - `rhyme_substitution` - `rhyme_content` - `era_corruption` - `meter_transformation` - `meter_destruction` - `meter_inconsistency` ### Metadata Distribution Highlights Top values reported in the paper: - Meter: Al-Tawil 20.31%, Al-Kamil 16.10%, Al-Basit 12.27% - Poet era: Modern 37.17%, Abbasid 22.59%, Mamluk 10.05% - Genre: General 22.89%, Short 11.61%, Praise 8.01% ## Dataset Structure ### Release Organization The Hugging Face dataset viewer is configured with four subsets so each template file appears separately in the subset dropdown: - `generation_templates` - `continuation_templates` - `analysis_templates` - `corruption_templates` These subsets map to the following files: - `generation_templates` -> `Templates - Poetry Generation.csv` - `continuation_templates` -> `Templates - Poetry Continuation.csv` - `analysis_templates` -> `Templates - Poetry Analysis.csv` - `corruption_templates` -> `Templates - Poetry Corruption.csv` At minimum, each example contains the columns provided in its source CSV file. ### Example Schema Generation / continuation / analysis template files include columns such as: - `Placeholder` - `Output` or `output` - `MSA` - `Nile Valley` - `North Africa` - `Gulf` - `Levant` The corruption template file includes columns such as: - `Placeholder` - `corruption_type` - `MSA` - `Nile Valley` - `North Africa` - `Gulf` - `Levant` ## Intended Uses ### Direct Use - Instruction tuning for Arabic poetry-capable language models - Evaluation of controllable Arabic poetry generation - Study of dialectal prompt robustness in Arabic - Research on meter-, rhyme-, style-, and metadata-conditioned generation - Benchmarking poetry analysis with multiple-choice supervisionpretty_name: 阿拉伯诗歌IFT（Arabic Poetry IFT） language: - 阿拉伯语 language_creators: - 公开采集 license: 其他 multilinguality: 多语言 size_categories: - 100万<n<1000万 source_datasets: - 原始数据集 task_categories: - 文本生成 - 问答 task_ids: - 文本到文本生成 - 多项选择问答 annotations_creators: - 专家生成 - 机器生成 tags: - 阿拉伯语 - 诗歌 - 指令遵循 - 方言 - 文学文本 configs: - config_name: 生成模板 default: true data_files: - split: 生成模板 path: "Templates - Poetry Generation.csv" - config_name: 续写模板 data_files: - split: 续写模板 path: "Templates - Poetry Continuation.csv" - config_name: 分析模板 data_files: - split: 分析模板 path: "Templates - Poetry Analysis.csv" - config_name: 篡改模板 data_files: - split: 篡改模板 path: "Templates - Poetry Corruption.csv" # 阿拉伯诗歌IFT（Arabic Poetry IFT） ## 数据集概览 阿拉伯诗歌IFT是一款面向阿拉伯诗歌理解与协同创作的大规模指令遵循数据集，涵盖四大任务家族：生成、续写、修正/复原与多项选择分析。该数据集支持现代标准阿拉伯语（Modern Standard Arabic, MSA），并在指令层使用了四种阿拉伯地域方言：海湾阿拉伯语、黎凡特阿拉伯语、尼罗河谷阿拉伯语与北非阿拉伯语。 本数据集配套发表于ACL 2026的论文《阿拉伯语及其方言的指令引导诗歌生成》（Instruction-Guided Poetry Generation in Arabic and Its Dialects）。 **数据集策展人：** Abdelrahman Sadallah、Kareem Elozeiri、Mervat Abassy、Rania Elbadry、Mohamed Anwar、Abed Alhakim Freihat、Preslav Nakov、Fajri Koto。 ### 支持任务 1. `生成（generation）`：根据标题、诗人、时代、体裁、格律、押韵、关键词或关键短语等约束创作诗歌 2. `续写（continuation）`：在保留诗歌格律约束的前提下续写未完成的诗作 3. `修正（revision）`/`篡改（corruption）`：将被篡改的诗歌恢复为原始合规形式 4. `分析（analysis）`：回答关于诗歌元数据的多项选择问题，涵盖诗人、标题、关键词、格律、时代、体裁与押韵规则 ### 语言覆盖范围 - 现代标准阿拉伯语（Modern Standard Arabic, MSA） - 海湾阿拉伯语 - 黎凡特阿拉伯语 - 尼罗河谷阿拉伯语 - 北非阿拉伯语 大部分源诗歌采用现代标准阿拉伯语，方言覆盖主要通过人工编写的指令模板实现，仅辅以少量方言源诗歌素材。 ## 数据集构建 ### 源数据采集 底层诗歌语料从公开文学资源聚合而来，经统一格式、元数据丰富与去重后用于指令生成。仅包含单节的诗歌被移除。当至少70%的诗节结尾经归一化后匹配时，将自动推断押韵规则。 论文中公布的训练语料统计如下： | 数据源 | 训练诗歌数 | 平均诗节数 | |---|---:|---:| | Ashaar | 123,581 | 19.81 | | PoetsGate* | 112,482 | 15.58 | | Adab* | 70,277 | 35.33 | | AraPoems | 62,963 | 22.01 | | Diwan* | 38,005 | 22.65 | | Mawsooaa* | 18,002 | 10.25 | | Arapoet* | 1,303 | 9.25 | | Arabic Poetry Dataset | 662 | 19.41 | | Arabic-Poetry-Melody | 48 | 21.44 | | Adab World* | 6 | 93.33 | | Other | 8 | 24.88 | | **总计** | **427,337** | **21.39** | 论文中公布的测试基准统计如下： | 数据源 | 测试诗歌数 | 平均诗节数 | |---|---:|---:| | FannOrFlop（Al Ghallabi et al., 2025） | 6,984 | 17.97 | *注：带*的数据源为爬取资源，引自原论文。 ### 元数据丰富 语料同时使用原始元数据与衍生元数据。用于构建指令任务的核心字段包括： - `poem_text`（诗歌文本） - `poem_title`（诗歌标题） - `poet_name`（诗人姓名） - `poet_era`（诗人时代） - `genre`（诗歌体裁） - `meter`（诗歌格律） - `rhyme`（押韵规则） - `keywords`（关键词） - `key_phrases`（关键短语） 其中`keywords`与`key_phrases`通过Gemini 2.5 Pro自动生成。论文报告对100份抽样诗歌进行了人工质量校验，结果显示96%的提取关键词质量合格。 ### 去重与数据泄露防控 - 归一化后移除源内部重复数据 - 正字法归一化包括移除长音符号、移除变音符号与标准化拼写变体 - 移除所有与FannOrFlop基准集重叠的诗歌，以避免训练集与测试集的数据泄露。 ### 指令模板构建 针对四大任务家族手动设计指令层，随后扩展至五种阿拉伯语变体。终稿论文公布的统计如下： | 任务家族 | 基础模板数 | 覆盖阿拉伯语变体数 | 方言专属模板数 | |---|---:|---:|---:| | 诗歌生成 | 246 | 5 | 1,230 | | 诗歌续写 | 176 | 5 | 880 | | 诗歌分析 | 214 | 5 | 1,070 | | 诗歌修正 | 8 | 5 | 40 | | **总计** | **644** | **5** | **3,220** | 方言模板由对应地域变体的母语使用者编写并修订。 ## 数据集统计 ### 整体指令微调数据集规模 本指令遵循数据集包含**1,350,897**条训练样本与**24,815**条测试样本，总计**1,375,712**条示例。 | 任务类型 | 训练样本数 | 测试样本数 | 训练子任务数 | 测试子任务数 | |---|---:|---:|---:|---:| | 诗歌生成 | 427,337 | 6,984 | 19 | 19 | | 诗歌续写 | 427,276 | 6,984 | 11 | 11 | | 诗歌修正 | 68,947 | 3,863 | 8 | 8 | | 诗歌分析 | 427,337 | 6,984 | 16 | 14 | ### 任务特定说明 - 续写样本通过在诗歌长度的10%至90%之间随机截断点拆分诗歌生成 - 分析任务被框架化为多项选择问答，包含1个正确答案与4个干扰项 - 修正样本通过将自动篡改后的诗歌与其干净原始版本配对生成 八种诗歌篡改类型包括： - `rhyme_structure`（押韵结构篡改） - `full_style`（整体风格篡改） - `rhyme_substitution`（押韵替换） - `rhyme_content`（押韵内容篡改） - `era_corruption`（时代信息篡改） - `meter_transformation`（格律转换） - `meter_destruction`（格律破坏） - `meter_inconsistency`（格律不一致） ### 元数据分布亮点 论文公布的Top值分布如下： - 格律：Al-Tawil（20.31%）、Al-Kamil（16.10%）、Al-Basit（12.27%） - 诗人时代：现代（37.17%）、阿拔斯王朝（22.59%）、马木留克王朝（10.05%） - 体裁：通用（22.89%）、短篇（11.61%）、赞美诗（8.01%） ## 数据集结构 ### 发布组织形式 Hugging Face数据集查看器配置了四个子集，每个模板文件均可在子集下拉菜单中单独查看： - `generation_templates`（生成模板子集） - `continuation_templates`（续写模板子集） - `analysis_templates`（分析模板子集） - `corruption_templates`（篡改模板子集） 各子集对应的文件如下： - `generation_templates` → `Templates - Poetry Generation.csv`（诗歌生成模板.csv） - `continuation_templates` → `Templates - Poetry Continuation.csv`（诗歌续写模板.csv） - `analysis_templates` → `Templates - Poetry Analysis.csv`（诗歌分析模板.csv） - `corruption_templates` → `Templates - Poetry Corruption.csv`（诗歌篡改模板.csv） 每个示例至少包含对应源CSV文件中的列字段。 ### 示例数据结构 生成、续写与分析模板文件包含如下列： - `Placeholder`（占位符） - `Output`或`output`（输出内容） - `MSA`（现代标准阿拉伯语） - `Nile Valley`（尼罗河谷阿拉伯语） - `North Africa`（北非阿拉伯语） - `Gulf`（海湾阿拉伯语） - `Levant`（黎凡特阿拉伯语） 篡改模板文件包含如下列： - `Placeholder`（占位符） - `corruption_type`（篡改类型） - `MSA`（现代标准阿拉伯语） - `Nile Valley`（尼罗河谷阿拉伯语） - `North Africa`（北非阿拉伯语） - `Gulf`（海湾阿拉伯语） - `Levant`（黎凡特阿拉伯语） ## 预期应用场景 ### 直接应用 - 面向具备阿拉伯诗歌生成能力的语言模型的指令微调 - 可控阿拉伯诗歌生成效果的评估 - 阿拉伯语方言提示鲁棒性研究 - 基于格律、押韵、风格与元数据约束的生成研究 - 基于多项选择监督的诗歌分析基准测试