tvbat/SciMDIX

--- dataset_info: - config_name: aspects features: - name: filename dtype: large_string - name: abstract dtype: large_string - name: aspect_annotation dtype: large_string - name: aspects dtype: large_string splits: - name: train_ru num_bytes: 1628735 num_examples: 186 - name: train_kz num_bytes: 1607806 num_examples: 186 - name: test_ru num_bytes: 167148 num_examples: 20 - name: test_kz num_bytes: 165328 num_examples: 20 download_size: 1639452 dataset_size: 3569017 - config_name: ner_re features: - name: filename dtype: large_string - name: abstract dtype: large_string - name: annotation dtype: large_string - name: entities dtype: large_string - name: relations dtype: large_string splits: - name: train_ru num_bytes: 1620694 num_examples: 186 - name: train_kz num_bytes: 1593191 num_examples: 186 - name: test_ru num_bytes: 163368 num_examples: 20 - name: test_kz num_bytes: 163073 num_examples: 20 download_size: 1564526 dataset_size: 3540326 configs: - config_name: aspects data_files: - split: train_ru path: aspects/train_ru-* - split: train_kz path: aspects/train_kz-* - split: test_ru path: aspects/test_ru-* - split: test_kz path: aspects/test_kz-* - config_name: ner_re data_files: - split: train_ru path: ner_re/train_ru-* - split: train_kz path: ner_re/train_kz-* - split: test_ru path: ner_re/test_ru-* - split: test_kz path: ner_re/test_kz-* --- # SciMDIX Dataset ## Dataset Description SciMDIX is a bilingual dataset containing scientific abstracts in Russian and Kazakh across four domains: **IT, Linguistics, Medicine, and Psychology**. It is designed for advanced Information Extraction tasks and is divided into two main configurations: 1. **ner_re**: Contains annotations for Named Entity Recognition (NER) and Relation Extraction (RE). 2. **aspects**: Contains aspect-level markup (AIM, MATERIAL, METHOD, RESULT, TASK, TOOL, USAGE) for the same texts. ## Data Structure Each configuration contains four splits: `train_ru`, `train_kz`, `test_ru`, and `test_kz`. *Note: You can identify the domain of a specific text by looking at the prefix in the `filename` column (`it-`, `ling-`, `med-`, `psy-`).* ### Configuration: `ner_re` - `filename`: Original text file name (includes domain prefix). - `abstract`: The raw text of the scientific abstract. - `annotation`: Text with inline BRAT-style markup `[Entity|ID|TYPE]`. - `entities`: Extracted entities in BRAT format. - `relations`: Extracted relations between entities. ### Configuration: `aspects` - `filename`: Original text file name (includes domain prefix). - `abstract`: The raw text of the scientific abstract. - `aspect_annotation`: Text with inline aspect markup `[Span|ID|TYPE]`. - `aspects`: Extracted aspects in BRAT format. ## How to use You can load the dataset using the `datasets` library. Specify the configuration (`ner_re` or `aspects`) and the split you want to use: ```python from datasets import load_dataset # Load the Russian Train split for NER and Relation Extraction ds_ner_ru_train = load_dataset("tvbat/SciMDIX", "ner_re", split="train_ru") print(ds_ner_ru_train[0]) # Load the Kazakh Test split for Aspects ds_asp_kz_test = load_dataset("tvbat/SciMDIX", "aspects", split="test_kz") ``` ## Repository The code, models, and additional resources related to this dataset can be found in our [GitHub repository](https://github.com/tvbat/sci-text-miner-scimdix/tree/main) ## Citation If you use the **SciMDIX** dataset in your research, please cite our papers: 1. Batura T., Yerimbetova A., Mukazhanov N., Shvarts N., Sakenov B., Turdalyuly M. [Information Extraction from Multi-Domain Scientific Documents: Methods and Insights](https://doi.org/10.3390/app15169086). Applied Sciences. MDPI. 2025. V.15, 9086. ```bibtex @article{scimdix2025, author = {Batura, Tatiana and Yerimbetova, Aigerim and Mukazhanov, Nurzhan and Shvarts, Nikita and Sakenov, Bakzhan and Turdalyuly, Mussa}, title = {Information Extraction from Multi-Domain Scientific Documents: Methods and Insights}, journal = {Applied Sciences}, volume = {15}, year = {2025}, number = {16}, article-number = {9086}, publisher = {MDPI}, doi = {https://doi.org/10.3390/app15169086} } ``` 2. Shvarts N., Batura T., Mukazhanov N., Yerimbetova A., Turdalyuly M., Sakenov B. [SciMDIX: A dataset for aspect extraction from multi-domain scientific documents in Kazakh and Russian](https://doi.org/10.1016/j.procs.2026.01.056). *Procedia Computer Science*. 2026. V. 275, pp.474-483. ```bibtex @article{scimdix2026, title={SciMDIX: A dataset for aspect extraction from multi-domain scientific documents in Kazakh and Russian}, author={Shvarts, Nikita and Batura, Tatiana and Mukazhanov, Nurzhan and Yerimbetova, Aigerim and Turdalyuly, Mussa and Sakenov, Bakzhan}, journal={Procedia Computer Science}, volume={275}, pages={474--483}, year={2026}, publisher={Elsevier}, doi = {https://doi.org/10.1016/j.procs.2026.01.056} } ```数据集元信息： - 配置名称：aspects 特征字段： - 字段名：filename，数据类型：大字符串 - 字段名：abstract，数据类型：大字符串 - 字段名：aspect_annotation，数据类型：大字符串 - 字段名：aspects，数据类型：大字符串 子集拆分： - 子集名称：train_ru，字节数：1628735，样本量：186 - 子集名称：train_kz，字节数：1607806，样本量：186 - 子集名称：test_ru，字节数：167148，样本量：20 - 子集名称：test_kz，字节数：165328，样本量：20 下载大小：1639452，数据集总大小：3569017 - 配置名称：ner_re 特征字段： - 字段名：filename，数据类型：大字符串 - 字段名：abstract，数据类型：大字符串 - 字段名：annotation，数据类型：大字符串 - 字段名：entities，数据类型：大字符串 - 字段名：relations，数据类型：大字符串 子集拆分： - 子集名称：train_ru，字节数：1620694，样本量：186 - 子集名称：train_kz，字节数：1593191，样本量：186 - 子集名称：test_ru，字节数：163368，样本量：20 - 子集名称：test_kz，字节数：163073，样本量：20 下载大小：1564526，数据集总大小：3540326 配置项： - 配置名称：aspects，数据文件： - 拆分：train_ru，路径：aspects/train_ru-* - 拆分：train_kz，路径：aspects/train_kz-* - 拆分：test_ru，路径：aspects/test_ru-* - 拆分：test_kz，路径：aspects/test_kz-* - 配置名称：ner_re，数据文件： - 拆分：train_ru，路径：ner_re/train_ru-* - 拆分：train_kz，路径：ner_re/train_kz-* - 拆分：test_ru，路径：ner_re/test_ru-* - 拆分：test_kz，路径：ner_re/test_kz-* # SciMDIX数据集 ## 数据集说明 SciMDIX是一款双语数据集，涵盖俄语与哈萨克语的科技摘要，共包含四个领域：**信息技术（Information Technology, IT）、语言学、医学与心理学**。本数据集面向高级信息抽取任务设计，分为两大配置分支： 1. **ner_re配置**：包含命名实体识别（Named Entity Recognition, NER）与关系抽取（Relation Extraction, RE）标注信息。 2. **aspects配置**：针对相同文本提供细粒度方面标注，涵盖AIM、MATERIAL、METHOD、RESULT、TASK、TOOL、USAGE七大类别。 ## 数据结构 每个配置分支均包含四个子集拆分：`train_ru`、`train_kz`、`test_ru`与`test_kz`。 *注：可通过`filename`字段的前缀标识特定文本所属领域：`it-`代表信息技术领域、`ling-`代表语言学领域、`med-`代表医学领域、`psy-`代表心理学领域。* ### 配置分支：ner_re - `filename`：原始文本文件名（包含领域前缀） - `abstract`：科技摘要的原始文本 - `annotation`：采用BRAT格式的行内标注文本，格式为`[Entity|ID|TYPE]` - `entities`：BRAT格式的抽取实体集合 - `relations`：实体间的抽取关系集合 ### 配置分支：aspects - `filename`：原始文本文件名（包含领域前缀） - `abstract`：科技摘要的原始文本 - `aspect_annotation`：采用行内格式的方面标注文本，格式为`[Span|ID|TYPE]` - `aspects`：BRAT格式的抽取方面集合 ## 使用方法 可通过`datasets`库加载本数据集，需指定配置分支（`ner_re`或`aspects`）与目标子集拆分： python from datasets import load_dataset # 加载用于命名实体识别与关系抽取任务的俄语训练子集 ds_ner_ru_train = load_dataset("tvbat/SciMDIX", "ner_re", split="train_ru") print(ds_ner_ru_train[0]) # 加载用于方面抽取任务的哈萨克语测试子集 ds_asp_kz_test = load_dataset("tvbat/SciMDIX", "aspects", split="test_kz") ## 代码仓库 本数据集相关的代码、模型与额外资源均可在我们的[GitHub仓库](https://github.com/tvbat/sci-text-miner-scimdix/tree/main)中获取。 ## 引用声明 若您在研究中使用**SciMDIX**数据集，请引用以下论文： 1. Batura T.、Yerimbetova A.、Mukazhanov N.、Shvarts N.、Sakenov B.、Turdalyuly M. 所著《Information Extraction from Multi-Domain Scientific Documents: Methods and Insights》，发表于*Applied Sciences*，MDPI出版社，2025年，第15卷，第9086篇文章。 bibtex @article{scimdix2025, author = {Batura, Tatiana and Yerimbetova, Aigerim and Mukazhanov, Nurzhan and Shvarts, Nikita and Sakenov, Bakzhan and Turdalyuly, Mussa}, title = {Information Extraction from Multi-Domain Scientific Documents: Methods and Insights}, journal = {Applied Sciences}, volume = {15}, year = {2025}, number = {16}, article-number = {9086}, publisher = {MDPI}, doi = {https://doi.org/10.3390/app15169086} } 2. Shvarts N.、Batura T.、Mukazhanov N.、Yerimbetova A.、Turdalyuly M.、Sakenov B. 所著《SciMDIX: A dataset for aspect extraction from multi-domain scientific documents in Kazakh and Russian》，发表于*Procedia Computer Science*，Elsevier出版社，2026年，第275卷，第474-483页。 bibtex @article{scimdix2026, title={SciMDIX: A dataset for aspect extraction from multi-domain scientific documents in Kazakh and Russian}, author={Shvarts, Nikita and Batura, Tatiana and Mukazhanov, Nurzhan and Yerimbetova, Aigerim and Turdalyuly, Mussa and Sakenov, Bakzhan}, journal={Procedia Computer Science}, volume={275}, pages={474--483}, year={2026}, publisher={Elsevier}, doi = {https://doi.org/10.1016/j.procs.2026.01.056} }