kalixlouiis/myX-Corpus

---
license: apache-2.0
task_categories:
- feature-extraction
- text-generation
language:
- my
- en
---
# Dataset Card for kalixlouiis/myX-Corpus

**myX-Corpus** is a high-quality, bilingual (Burmese-English) dataset synthesized primarily for training the **myX-Tokenizer** series. By combining a cleaned Burmese corpus with high-quality English sentences, this dataset provides a balanced linguistic foundation for Natural Language Processing tasks in code-mixed environments.

## Dataset Summary

The dataset is a combination of 2.3 million Burmese sentences and approximately 350,000 sampled English sentences. While its primary purpose is tokenizer training, the corpus is designed to be versatile and can be utilized for Language Modeling, Text Classification, or any general-purpose NLP research involving the Burmese language.

## Data Synthesis Logic

The corpus was generated by merging two primary datasets with specific sampling logic:

1. **Burmese Base:** Integrated the complete cleaned corpus from [kalixlouiis/raw-data](https://huggingface.co/datasets/kalixlouiis/raw-data).
2. **English Enrichment:** Sampled 350,000 sentences from [agentlans/high-quality-english-sentences](https://huggingface.co/datasets/agentlans/high-quality-english-sentences).
3. **Filtering:** English sentences were filtered based on length (between 30 and 200 characters) to ensure the training data remained meaningful and concise.
4. **Shuffle:** The combined sentences were randomized to ensure a balanced distribution of languages throughout the corpus.

## Source Credits

We gratefully acknowledge the following sources used to build this corpus:

* **[kalixlouiis/raw-data](https://huggingface.co/datasets/kalixlouiis/raw-data):** Provided the cleaned, multi-domain Burmese foundational text.
* **[agentlans/high-quality-english-sentences](https://huggingface.co/datasets/agentlans/high-quality-english-sentences):** Provided the high-quality English sentence base used for bilingual optimization.

## Dataset Quality & Characteristics

* **Bilingual Balance:** The inclusion of 350,000 English rows ensures that models trained on this data (like myX-Tokenizer) can handle English text efficiently without excessive subword fragmentation.
* **Cleanliness:** Inherits the rigorous cleaning standards of the raw-data base, including the removal of Zawgyi encoding and Unicode errors.
* **Diversity:** Covers news, literature, conversational data, and general knowledge.

## Use Cases

While created for the **myX-Tokenizer** series, you are encouraged to use this dataset for:

* **Large Language Model (LLM) Pre-training:** For models targeting the Myanmar market.
* **Bilingual Embeddings:** Generating vector representations that understand both Burmese and English contexts.
* **Code-Switching Research:** Analyzing how Burmese and English interact in modern digital text.

## Licensing

This dataset is released for open-source research. Users should also respect the licensing terms of the original source datasets cited above.