KoseiUemura/KinNewsClassification

--- annotations_creators: - expert-annotated language: - kin - run license: mit multilinguality: multilingual source_datasets: - andreniyongabo/kinnews_kirnews task_categories: - text-classification task_ids: - topic-classification dataset_info: - config_name: kinnews_cleaned features: - name: text dtype: string - name: label dtype: int64 splits: - name: train num_bytes: 20011123 num_examples: 9075 - name: test num_bytes: 4432204 num_examples: 2008 download_size: 13396388 dataset_size: 24443327 - config_name: kirnews_cleaned features: - name: text dtype: string - name: label dtype: int64 splits: - name: train num_bytes: 2828871 num_examples: 1749 - name: test num_bytes: 87818 num_examples: 65 download_size: 1629277 dataset_size: 2916689 configs: - config_name: kinnews_cleaned data_files: - split: train path: kinnews_cleaned/train-* - split: test path: kinnews_cleaned/test-* - config_name: kirnews_cleaned data_files: - split: train path: kirnews_cleaned/train-* - split: test path: kirnews_cleaned/test-* tags: - mteb - text --- <!-- adapted from https://github.com/huggingface/huggingface_hub/blob/v0.30.2/src/huggingface_hub/templates/datasetcard_template.md --> <div align="center" style="padding: 40px 20px; background-color: white; border-radius: 12px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05); max-width: 600px; margin: 0 auto;"> <h1 style="font-size: 3.5rem; color: #1a1a1a; margin: 0 0 20px 0; letter-spacing: 2px; font-weight: 700;">KinNewsClassification</h1> <div style="font-size: 1.5rem; color: #4a4a4a; margin-bottom: 5px; font-weight: 300;">An <a href="https://github.com/embeddings-benchmark/mteb" style="color: #2c5282; font-weight: 600; text-decoration: none;" onmouseover="this.style.textDecoration='underline'" onmouseout="this.style.textDecoration='none'">MTEB</a> dataset</div> <div style="font-size: 0.9rem; color: #2c5282; margin-top: 10px;">Massive Text Embedding Benchmark</div> </div> Kinyarwanda and Kirundi news classification datasets (KINNEWS and KIRNEWS, respectively), which were both collected from Rwanda and Burundi news websites and newspapers, for low-resource monolingual and cross-lingual multiclass classification tasks. Each news article can be classified into one of 14 possible classes: politics, sport, economy, health, entertainment, history, technology, culture, religion, environment, education, relationship. | | | |---------------|---------------------------------------------| | Task category | t2c | | Domains | News, Written | | Reference | https://arxiv.org/abs/2010.12174 | Source datasets: - [andreniyongabo/kinnews_kirnews](https://huggingface.co/datasets/andreniyongabo/kinnews_kirnews) ## Dataset Preparation in MTEB This repository is a staging copy of `andreniyongabo/kinnews_kirnews` for MTEB. The intended long-term canonical benchmark copy is `mteb/KinNewsClassification`. ### Transformations - Built `text = title + " " + content` - Preserved the two source configs: `kinnews_cleaned` and `kirnews_cleaned` - Applied dataset cleaning before upload to reduce duplicates and train-test leakage in the benchmark copy ### Label Schema - Integer class labels from the source dataset are preserved ### Splits and subsets - `kinnews_cleaned`: cleaned train/test splits - `kirnews_cleaned`: cleaned train/test splits ## How to evaluate on this task You can evaluate an embedding model on this dataset using the following code: ```python import mteb task = mteb.get_task("KinNewsClassification") evaluator = mteb.MTEB([task]) model = mteb.get_model(YOUR_MODEL) evaluator.run(model) ``` <!-- Datasets want link to arxiv in readme to autolink dataset with paper --> To learn more about how to run models on `mteb` task check out the [GitHub repository](https://github.com/embeddings-benchmark/mteb). ## Citation If you use this dataset, please cite the dataset as well as [mteb](https://github.com/embeddings-benchmark/mteb), as this dataset likely includes additional processing as a part of the [MMTEB Contribution](https://github.com/embeddings-benchmark/mteb/tree/main/docs/mmteb). ```bibtex @article{niyongabo2020kinnews, author = {Niyongabo, Rubungo Andre and Qu, Hong and Kreutzer, Julia and Huang, Li}, journal = {arXiv preprint arXiv:2010.12174}, title = {KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi}, year = {2020}, } @article{enevoldsen2025mmtebmassivemultilingualtext, title={MMTEB: Massive Multilingual Text Embedding Benchmark}, author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff}, publisher = {arXiv}, journal={arXiv preprint arXiv:2502.13595}, year={2025}, url={https://arxiv.org/abs/2502.13595}, doi = {10.48550/arXiv.2502.13595}, } @article{muennighoff2022mteb, author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils}, title = {MTEB: Massive Text Embedding Benchmark}, publisher = {arXiv}, journal={arXiv preprint arXiv:2210.07316}, year = {2022} url = {https://arxiv.org/abs/2210.07316}, doi = {10.48550/ARXIV.2210.07316}, } ``` # Dataset Statistics <details> <summary> Dataset Statistics</summary> The following code contains the descriptive statistics from the task. These can also be obtained using: ```python import mteb task = mteb.get_task("KinNewsClassification") desc_stats = task.metadata.descriptive_stats ``` ```json { "test": { "num_samples": 5177, "number_texts_intersect_with_train": 1294, "text_statistics": { "total_text_length": 9500557, "min_text_length": 137, "average_text_length": 1835.1471894919837, "max_text_length": 32815, "unique_texts": 3367 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 14, "labels": { "1": { "count": 926 }, "10": { "count": 162 }, "3": { "count": 408 }, "4": { "count": 680 }, "0": { "count": 1311 }, "6": { "count": 101 }, "2": { "count": 971 }, "8": { "count": 108 }, "13": { "count": 200 }, "9": { "count": 43 }, "12": { "count": 78 }, "11": { "count": 45 }, "5": { "count": 67 }, "7": { "count": 77 } } }, "hf_subset_descriptive_stats": { "kinnews_cleaned": { "num_samples": 4254, "number_texts_intersect_with_train": 678, "text_statistics": { "total_text_length": 7979747, "min_text_length": 137, "average_text_length": 1875.822049835449, "max_text_length": 32815, "unique_texts": 2686 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 14, "labels": { "1": { "count": 635 }, "10": { "count": 154 }, "3": { "count": 366 }, "4": { "count": 651 }, "0": { "count": 979 }, "6": { "count": 87 }, "2": { "count": 899 }, "8": { "count": 95 }, "13": { "count": 133 }, "9": { "count": 43 }, "12": { "count": 34 }, "11": { "count": 36 }, "5": { "count": 65 }, "7": { "count": 77 } } } }, "kirnews_cleaned": { "num_samples": 923, "number_texts_intersect_with_train": 616, "text_statistics": { "total_text_length": 1520810, "min_text_length": 153, "average_text_length": 1647.6814734561212, "max_text_length": 6449, "unique_texts": 681 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 12, "labels": { "1": { "count": 291 }, "0": { "count": 332 }, "13": { "count": 67 }, "12": { "count": 44 }, "2": { "count": 72 }, "8": { "count": 13 }, "4": { "count": 29 }, "10": { "count": 8 }, "3": { "count": 42 }, "11": { "count": 9 }, "6": { "count": 14 }, "5": { "count": 2 } } } } } }, "train": { "num_samples": 20703, "number_texts_intersect_with_train": null, "text_statistics": { "total_text_length": 38259142, "min_text_length": 33, "average_text_length": 1847.999903395643, "max_text_length": 67385, "unique_texts": 10824 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 14, "labels": { "2": { "count": 3835 }, "4": { "count": 2685 }, "1": { "count": 3578 }, "6": { "count": 408 }, "0": { "count": 5239 }, "10": { "count": 668 }, "11": { "count": 226 }, "3": { "count": 1770 }, "9": { "count": 162 }, "8": { "count": 380 }, "7": { "count": 301 }, "13": { "count": 928 }, "12": { "count": 314 }, "5": { "count": 209 } } }, "hf_subset_descriptive_stats": { "kinnews_cleaned": { "num_samples": 17014, "number_texts_intersect_with_train": null, "text_statistics": { "total_text_length": 31840528, "min_text_length": 33, "average_text_length": 1871.431056776772, "max_text_length": 67385, "unique_texts": 9075 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 14, "labels": { "2": { "count": 3519 }, "4": { "count": 2578 }, "1": { "count": 2469 }, "6": { "count": 352 }, "0": { "count": 3929 }, "10": { "count": 628 }, "11": { "count": 177 }, "3": { "count": 1548 }, "9": { "count": 162 }, "8": { "count": 341 }, "7": { "count": 301 }, "13": { "count": 667 }, "12": { "count": 152 }, "5": { "count": 191 } } } }, "kirnews_cleaned": { "num_samples": 3689, "number_texts_intersect_with_train": null, "text_statistics": { "total_text_length": 6418614, "min_text_length": 117, "average_text_length": 1739.9333152615884, "max_text_length": 10925, "unique_texts": 1749 }, "image_statistics": null, "audio_statistics": null, "label_statistics": { "min_labels_per_text": 1, "average_label_per_text": 1.0, "max_labels_per_text": 1, "unique_labels": 12, "labels": { "0": { "count": 1310 }, "1": { "count": 1109 }, "3": { "count": 222 }, "4": { "count": 107 }, "2": { "count": 316 }, "10": { "count": 40 }, "6": { "count": 56 }, "8": { "count": 39 }, "5": { "count": 18 }, "13": { "count": 261 }, "12": { "count": 162 }, "11": { "count": 49 } } } } } } } ``` </details> --- *This dataset card was automatically generated using [MTEB](https://github.com/embeddings-benchmark/mteb)*annotations_creators: 专家标注 languages: 基尼亚卢旺达语（kin）、基隆迪语（run） license: MIT协议 multilinguality: 多语言 source_datasets: andreniyongabo/kinnews_kirnews task_categories: 文本分类 task_ids: 主题分类 dataset_info: - config_name: kinnews_cleaned features: - name: 文本 dtype: 字符串 - name: 标签 dtype: int64 splits: - name: 训练集 num_bytes: 20011123 num_examples: 9075 - name: 测试集 num_bytes: 4432204 num_examples: 2008 - config_name: kirnews_cleaned features: - name: 文本 dtype: 字符串 - name: 标签 dtype: int64 splits: - name: 训练集 num_bytes: 2828871 num_examples: 1749 - name: 测试集 num_bytes: 87818 num_examples: 65 download_size: 13396388 total_dataset_size: 24443327 <div align="center" style="padding: 40px 20px; background-color: white; border-radius: 12px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05); max-width: 600px; margin: 0 auto;"> <h1 style="font-size: 3.5rem; color: #1a1a1a; margin: 0 0 20px 0; letter-spacing: 2px; font-weight: 700;">KinNewsClassification</h1> <div style="font-size: 1.5rem; color: #4a4a4a; margin-bottom: 5px; font-weight: 300;">一个<a href="https://github.com/embeddings-benchmark/mteb" style="color: #2c5282; font-weight: 600; text-decoration: none;" onmouseover="this.style.textDecoration='underline'" onmouseout="this.style.textDecoration='none'">大规模文本嵌入基准（Massive Text Embedding Benchmark, MTEB）</a>数据集</div> <div style="font-size: 0.9rem; color: #2c5282; margin-top: 10px;">大规模文本嵌入基准</div> </div> 基尼亚卢旺达语与基隆迪语新闻分类数据集（分别对应KINNEWS与KIRNEWS），二者均采集自卢旺达与布隆迪的新闻网站及报纸，面向低资源单语与跨语言多类别分类任务。每篇新闻可被归类至以下14个类别：政治、体育、经济、健康、娱乐、历史、科技、文化、宗教、环境、教育、人际关系。 | | | |---------------|---------------------------------------------| | 任务类别 | 文本分类（t2c） | | 领域 | 新闻、书面文本 | | 参考文献 | https://arxiv.org/abs/2010.12174 | ### 源数据集 - [andreniyongabo/kinnews_kirnews](https://huggingface.co/datasets/andreniyongabo/kinnews_kirnews) ## MTEB数据集准备 本仓库为`andreniyongabo/kinnews_kirnews`针对MTEB的暂存副本，长期标准基准副本为`mteb/KinNewsClassification`。 ### 数据变换 - 构建文本字段：`text = 标题 + " " + 正文内容` - 保留两个源配置：`kinnews_cleaned`与`kirnews_cleaned` - 上传至基准平台前已完成数据清洗，以减少重复数据与训练测试集泄露问题 ### 标签规范 - 保留源数据集的整数类别标签 ### 数据拆分与子集 - `kinnews_cleaned`: 已清洗的训练/测试拆分 - `kirnews_cleaned`: 已清洗的训练/测试拆分 ## 任务评估方法 你可以通过以下代码在该数据集上评估嵌入模型： python import mteb task = mteb.get_task("KinNewsClassification") evaluator = mteb.MTEB([task]) model = mteb.get_model(YOUR_MODEL) evaluator.run(model) 如需了解如何在MTEB任务上运行模型，请访问[MTEB官方GitHub仓库](https://github.com/embeddings-benchmark/mteb)。 ## 引用说明 若使用本数据集，请同时引用原始数据集与MTEB，因为本数据集作为MMTEB贡献的一部分进行了额外处理。 bibtex @article{niyongabo2020kinnews, author = {Niyongabo, Rubungo Andre and Qu, Hong and Kreutzer, Julia and Huang, Li}, journal = {arXiv preprint arXiv:2010.12174}, title = {KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi}, year = {2020}, } @article{enevoldsen2025mmtebmassivemultilingualtext, title={MMTEB: Massive Multilingual Text Embedding Benchmark}, author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff}, publisher = {arXiv}, journal={arXiv preprint arXiv:2502.13595}, year={2025}, url={https://arxiv.org/abs/2502.13595}, doi = {10.48550/arXiv.2502.13595}, } @article{muennighoff2022mteb, author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils}, title = {MTEB: Massive Text Embedding Benchmark}, publisher = {arXiv}, journal={arXiv preprint arXiv:2210.07316}, year = {2022}, url = {https://arxiv.org/abs/2210.07316}, doi = {10.48550/ARXIV.2210.07316}, } ## 数据集统计信息 <details> <summary> 数据集统计详情</summary> 训练集与测试集的字节数与样本数： - kinnews_cleaned: - 训练集：20011123字节，9075个样本 - 测试集：4432204字节，2008个样本 - kirnews_cleaned: - 训练集：2828871字节，1749个样本 - 测试集：87818字节，65个样本 总数据集大小：24443327字节 </details> *本数据集卡片由[MTEB](https://github.com/embeddings-benchmark/mteb)自动生成*