AraSEG

AraSeg 是一个面向现代标准阿拉伯语（MSA）句子切分的手工标注、跨体裁基准数据集。其元数据标注语言为阿拉伯语，许可证为 MIT，size_categories 为 1K<n<10K，task_categories 为 token-classification，标签为 sentence-segmentation。数据集包含 2,187 篇文档、48,250 个段落、121,623 个句子，约 1.45M tokens，来自 30 个来源，覆盖 8 个体裁：Children、Education、Literature、Media、Poetry、Politics、Religion 和 Wiki。阿拉伯语标点稀疏、不一致且存在歧义，许多句子边界没有标点标记；AraSeg 中 39% 的句子边界是基于词边界的。因此数据集覆盖有标点/无标点、有段落/无段落等现实条件。大部分 gold 句子切分来自 CAMeL Treebank 和 BAREC 语料库，并加入完整 Quran 以及额外的 Old Testament 和 New Testament 文本以拓宽宗教体裁。任务定义为二分类 token classification：对每个 token 预测其后是否跟随句子边界。数据集提供四个 config 变体，区别在于标点和段落边界可用性：PA（默认）保留段落边界和标点；NP 不保留段落边界但保留标点；NoPnx-PA 保留段落边界但移除所有标点和符号字符，以及特定来源的边界标记（如 Quran、Old/New Testament 的节号，ArabicMMLU 的答案标签如 ( أ )）；NoPnx-NP 不保留段落边界且移除标点。PA 中以换行 token 表示段落边界；NP 中每篇文档为一个 token 流。可通过 datasets.load_dataset 加载 MBZUAI/AraSEG 的 NoPnx-NP 等配置。数据结构上，每行对应一篇文档，字段包括 doc_id、doc_name、source、genre、tokens、labels、paragraph_ids、sent_ids、text、label_str。tokens 是空白分词后的 token，标点已与词分开；在 PA 配置中，换行 token 标记段落边界。labels 为 1 表示该 token 之后句子结束，否则为 0，换行 token 始终为 0。paragraph_ids 是每个 token 的段落 id；sent_ids 是 token 在其段落内的句子 id，换行 token 为 -1；text 是 tokens 用单空格连接；label_str 是 labels 的 0/1 字符串，用于 shared-task 提交格式。source 例如 Majed、Hindawi、Quran、ArabicMMLU；genre 为 8 类之一。数据划分包括 Train 约 80%、Dev 约 10%、Test 约 10%，按词数划分，且在文档级别划分以避免泄漏。原始 CAMeL Treebank 和 BAREC 划分被保留；新增 Quran、Old Testament、New Testament 文档按词数贪心分配以平衡。四个 config 在每个 split 中包含相同文档，各 config 的 train/dev/test 文档数均为 1703/222/262。各 config 的数据集规模约为：PA dataset_size 67,217,904 字节、download_size 12,565,904 字节；NP 65,743,888/12,394,241；NoPnx-PA 57,604,474/11,894,149；NoPnx-NP 56,210,202/11,666,727。体裁统计：Children 422 docs、9.8K paras、16.7K sents、172.7K tokens、Pnx Density 20.6%、Avg. Sent. Len. 10.4；Education 605、18.8K、50.3K、373.3K、25.3%、7.4；Literature 406、8.0K、21.0K、339.4K、14.2%、16.2；Media 101、1.2K、2.5K、32.9K、12.4%、13.2；Poetry 30、1.7K、1.7K、11.7K、7.7%、6.9；Politics 17、5.9K、9.7K、147.0K、10.6%、15.1；Religion 438、1.4K、13.7K、216.0K、16.1%、15.7；Wiki 168、1.6K、6.1K、128.6K、13.8%、21.2。标注方面，AraSeg 复用 CAMeL Treebank 和 BAREC 的 gold sentence segmentation，遵循 CAMeL Treebank 指南：先根据标点（逗号、句号、分号、问号、感叹号、冒号）自动切分，再通过受控的 split 和 merge 人工修正；依赖从句或不完整从句从不切分为独立句子。对于 Quran、Old/New Testament 和诗歌，诗句边界即句子边界；在诗歌中诗句边界也是段落边界。对于没有显式段落的来源，段落遵循文档结构（如 Quran 章节、ArabicMMLU 问题）。在 ArabicMMLU 中，问题与每个答案选项分别是独立句子。评测方面，句子切分作为二分类 token classification，采用 Boundary Precision、Boundary Recall 和 Boundary F1。预测仅在与 gold 边界完全匹配时才算正确；分数按文档计算并对文档宏平均，以避免长文档主导。README 还提供引用信息，若使用 AraSeg 需引用相关 EMNLP 2026 Findings 论文。AraSeg is a manually annotated, cross-genre benchmark dataset for Modern Standard Arabic (MSA) sentence segmentation. Its metadata annotation language is Arabic, its license is MIT, its size_categories is 1K<n<10K, its task_categories is token-classification, and its label is sentence-segmentation. The dataset contains 2,187 documents, 48,250 paragraphs, 121,623 sentences, and about 1.45M tokens, drawn from 30 sources and covering 8 genres: Children, Education, Literature, Media, Poetry, Politics, Religion, and Wiki. Arabic punctuation is sparse, inconsistent, and ambiguous, and many sentence boundaries have no punctuation marks; 39% of sentence boundaries in AraSeg are word-boundary based. Therefore, the dataset covers realistic conditions such as with/without punctuation and with/without paragraphs. Most gold sentence segmentations come from the CAMeL Treebank and BAREC corpora, with the complete Quran and additional Old Testament and New Testament texts added to broaden the religious genre. The task is defined as binary token classification: for each token, predict whether a sentence boundary follows it. The dataset provides four config variants differing in punctuation and paragraph boundary availability: PA (default) keeps paragraph boundaries and punctuation; NP does not keep paragraph boundaries but keeps punctuation; NoPnx-PA keeps paragraph boundaries but removes all punctuation and symbol characters, as well as source-specific boundary markers (such as verse numbers in Quran and Old/New Testament, and answer labels in ArabicMMLU such as ( أ )); NoPnx-NP does not keep paragraph boundaries and removes punctuation. In PA, newline tokens represent paragraph boundaries; in NP, each document is a token stream. Configurations such as NoPnx-NP of MBZUAI/AraSEG can be loaded via datasets.load_dataset. Structurally, each row corresponds to one document, and fields include doc_id, doc_name, source, genre, tokens, labels, paragraph_ids, sent_ids, text, and label_str. tokens are whitespace-tokenized tokens with punctuation separated from words; in the PA configuration, newline tokens mark paragraph boundaries. labels is 1 if a sentence ends after that token, otherwise 0, and newline tokens are always 0. paragraph_ids is the paragraph id of each token; sent_ids is the sentence id of a token within its paragraph, with -1 for newline tokens; text is tokens joined by a single space; label_str is the 0/1 string form of labels for shared-task submission format. source examples include Majed, Hindawi, Quran, and ArabicMMLU; genre is one of the 8 classes. Data splits include Train about 80%, Dev about 10%, and Test about 10%, split by word count and at the document level to avoid leakage. Original CAMeL Treebank and BAREC splits are preserved; newly added Quran, Old Testament, and New Testament documents are greedily assigned by word count for balance. The four configs contain the same documents in each split, and each config has train/dev/test document counts of 1703/222/262. Approximate dataset sizes for each config are: PA dataset_size 67,217,904 bytes and download_size 12,565,904 bytes; NP 65,743,888/12,394,241; NoPnx-PA 57,604,474/11,894,149; NoPnx-NP 56,210,202/11,666,727. Genre statistics: Children 422 docs, 9.8K paras, 16.7K sents, 172.7K tokens, Pnx Density 20.6%, Avg. Sent. Len. 10.4; Education 605, 18.8K, 50.3K, 373.3K, 25.3%, 7.4; Literature 406, 8.0K, 21.0K, 339.4K, 14.2%, 16.2; Media 101, 1.2K, 2.5K, 32.9K, 12.4%, 13.2; Poetry 30, 1.7K, 1.7K, 11.7K, 7.7%, 6.9; Politics 17, 5.9K, 9.7K, 147.0K, 10.6%, 15.1; Religion 438, 1.4K, 13.7K, 216.0K, 16.1%, 15.7; Wiki 168, 1.6K, 6.1K, 128.6K, 13.8%, 21.2. For annotation, AraSeg reuses gold sentence segmentation from CAMeL Treebank and BAREC and follows CAMeL Treebank guidelines: first automatically split by punctuation (comma, period, semicolon, question mark, exclamation mark, colon), then manually correct through controlled split and merge; dependent clauses or incomplete clauses are never split into separate sentences. For Quran, Old/New Testament, and poetry, verse boundaries are sentence boundaries; in poetry, verse boundaries are also paragraph boundaries. For sources without explicit paragraphs, paragraphs follow document structure (such as Quran chapters and ArabicMMLU questions). In ArabicMMLU, the question and each answer option are separate sentences. For evaluation, sentence segmentation is treated as binary token classification and uses Boundary Precision, Boundary Recall, and Boundary F1. A prediction is correct only if it exactly matches a gold boundary; scores are computed per document and macro-averaged over documents to avoid long documents dominating. The README also provides citation information: if using AraSeg, cite the related EMNLP 2026 Findings paper.