ndamulelonemakh/zabantu-news

---
language:
- ve
license: cc-by-nc-sa-4.0
size_categories:
- 1K<n<10K
pretty_name: ZaBantu News Headlines[RAW]
dataset_info:
features:
- name: id
dtype: int64
- name: text
dtype: string
- name: language
dtype: string
- name: word_count
dtype: int64
- name: char_count
dtype: int64
- name: sentence_count
dtype: int64
- name: PotentialSplits
sequence: string
- name: md5
dtype: string
splits:
- name: train
num_bytes: 8799449
num_examples: 16607
download_size: 5317813
dataset_size: 8799449
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
---

# ZaBantu News[Raw]

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## 1. Dataset Description

### Dataset Summary
ZaBantu News[Raw] is a collection of news headlines in Sepedi (Northern Sotho) and Tshivenda languages, sourced from various public media platforms. This dataset is designed to support NLP research and development in low-resource languages, providing raw, unlabelled headlines. It complements the "ZaBantu News" dataset, which includes a mix of machine and human labels following [IPTC news codes](https://www.iptc.org/std/NewsCodes/treeview/mediatopic/mediatopic-en-GB.html).

### Supported Tasks and Leaderboards
This dataset is suitable for tasks such as language modeling, unsupervised learning, and pre-training for downstream NLP tasks (e.g., text classification, topic modeling). It can also be used for linguistic studies and developing NLP tools for Sepedi and Tshivenda.

### Languages
The dataset contains text in Sepedi (Northern Sotho) and Tshivenda, both of which are low-resource languages spoken in South Africa.

## 2. Dataset Structure

### Data Instances
A data instance in "ZaBantu News[Raw]" consists of a single news headline. Some headlines may be segmented using delimiters like asterisks or full stops for further breakdown.

### Data Fields
- `headline`: the news headline text.
- `language`: language of the headline (Sepedi or Tshivenda).

### Data Splits
The dataset is currently not split into standard subsets (e.g., training, validation, testing). Users may consider splitting the dataset according to their needs for specific tasks.

## 3. Dataset Creation

### Curation Rationale
The dataset was curated to address the scarcity of digital resources in Sepedi and Tshivenda for NLP research. It aims to facilitate the development of language models and other NLP tools for these languages.

### Source Data
#### Initial Data Collection and Normalization
Headlines were collected from various public media platforms, ensuring a diverse representation of topics and styles.

#### Who are the source language producers?
The source language producers include journalists and content creators from multiple media outlets that publish news in Sepedi and Tshivenda.

### Annotations
#### Annotation process
The dataset is currently not annotated. It serves as a raw collection of news headlines for potential use in unsupervised learning tasks or as a foundation for future annotated datasets.

#### Who are the annotators?
Not applicable.

## 4. Motivation

- **Social Impact of Dataset** - The dataset contributes to NLP research in low-resource languages, potentially aiding in the development of technology that supports linguistic diversity and inclusion.

## 5. Licensing Information
The dataset is shared under a Share Alike license, allowing for adaptation and distribution of derivative works under the same or a compatible license.

## 6. Citation Information
Please cite this dataset as follows:

```bibtext
@misc{zabantu_news_raw,
author = {Ndamulelo Nemakhavhani},
title = {ZaBantu News[Raw]: Collecting Tshivenda and Sepedi News Headlines for Cross-Lingual Transfer Evaluation,
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://huggingface.co/datasets/ndamulelonemakh/zabantu-news}}
}
```

原始信息汇总

ZaBantu News[Raw] 数据集概述

1. 数据集描述

数据集摘要

ZaBantu News[Raw] 是一个包含 Sepedi（北索托语）和 Tshivenda 语言新闻标题的集合，来源于各种公共媒体平台。该数据集旨在支持低资源语言的 NLP 研究和开发，提供原始、未标记的标题。它补充了包含机器和人工标签的 "ZaBantu News" 数据集。

支持的任务和排行榜

该数据集适用于语言建模、无监督学习和下游 NLP 任务（如文本分类、主题建模）的预训练。它还可用于语言学研究和开发针对 Sepedi 和 Tshivenda 的 NLP 工具。

语言

数据集包含 Sepedi（北索托语）和 Tshivenda 文本，这两种语言都是南非的低资源语言。

2. 数据集结构

数据实例

"ZaBantu News[Raw]" 中的数据实例由单个新闻标题组成。一些标题可能使用星号或句号等分隔符进行进一步细分。

数据字段

• headline：新闻标题文本。
• language：标题的语言（Sepedi 或 Tshivenda）。

数据分割

数据集目前未分成标准子集（如训练、验证、测试）。用户可根据特定任务的需求自行分割数据集。

3. 数据集创建

策划理由

该数据集是为了解决 Sepedi 和 Tshivenda 在 NLP 研究中数字资源的稀缺问题而策划的。它旨在促进这些语言的语言模型和其他 NLP 工具的开发。

源数据

初始数据收集和规范化

标题从各种公共媒体平台收集，确保了主题和风格的多样性。

源语言生产者

源语言生产者包括来自多个媒体机构的 Sepedi 和 Tshivenda 新闻的记者和内容创作者。

注释

注释过程

数据集目前未注释。它作为新闻标题的原始集合，可用于无监督学习任务或作为未来注释数据集的基础。

注释者

不适用。

4. 动机

• 数据集的社会影响 - 该数据集有助于低资源语言的 NLP 研究，可能促进支持语言多样性和包容性的技术发展。

5. 许可信息

数据集在 Share Alike 许可下共享，允许在相同或兼容许可下改编和分发衍生作品。

6. 引用信息

请按以下方式引用此数据集：

bibtext @misc{zabantu_news_raw, author = {Ndamulelo Nemakhavhani}, title = {ZaBantu News[Raw]: Collecting Tshivenda and Sepedi News Headlines for Cross-Lingual Transfer Evaluation}, year = {2024}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {url{https://huggingface.co/datasets/ndamulelonemakh/zabantu-news}} }