hon9kon9ize/yue-sts

---
dataset_info:
features:
- name: text1
dtype: string
- name: text2
dtype: string
- name: label
dtype: int64
splits:
- name: train
num_bytes: 14030884.8
num_examples: 32103
- name: test
num_bytes: 1558987.2
num_examples: 3567
download_size: 10514888
dataset_size: 15589872
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: test
path: data/test-*
license: apache-2.0
task_categories:
- text-classification
tags:
- cantonese
- yue
- sts
pretty_name: Cantonese Wikipedia Semantic Textual Similarity Dataset
size_categories:
- 1K<n<10K
---

# Cantonese Semantic Textual Similarity Dataset

## Dataset Description

This dataset is generated by Gemini Flash 1.5 with [R5dwMg/zh-wiki-yue-long](https://huggingface.co/datasets/R5dwMg/zh-wiki-yue-long), which is a Cantonese Wikipedia dataset. We randomly selected a text from R5dwMg/zh-wiki-yue-long and feed it into the Gemini Flash 1.5 to generate and a positve and negative abstruct / question pair. The dataset is then split into training and testing set. The training set contains 32103 examples and the testing set contains 3567 examples. This dataset has been re-balance to have 50% positive and 50% negative examples.

## Licensing and Permissions

This dataset is derived from Wikipedia content. Please ensure compliance with Wikipedia's licensing and terms of use, particularly the Creative Commons Attribution-ShareAlike license (CC BY-SA). Proper attribution must be given for all content derived from Wikipedia.
Acknowledgments

We acknowledge Wikipedia and its contributors for providing the source content for this dataset.
Limitations and Considerations

- Content Variability: The dataset may contain a mix of different topics and contexts, as it is sourced from various Wikipedia articles.
- Complexity: The focus on long sentences could present challenges in parsing and processing complex structures.
- Ethical Considerations: Ensure appropriate use of the dataset in accordance with Wikipedia's licensing and any applicable data protection regulations.