kalixlouiis/raw-data

---
license: apache-2.0
task_categories:
- feature-extraction
- text-generation
- text-classification
language:
- my
- en
size_categories:
- 1M<n<10M
---
# Dataset Card for kalixlouiis/raw-data

This dataset is a high-quality, large-scale Burmese language corpus curated from a diverse range of sources, including classical literature, news, encyclopedic content, and conversational data. It has been rigorously cleaned to ensure linguistic integrity and is suitable for various NLP tasks such as Language Modeling, Tokenizer Training, and Text Classification.

## Dataset Summary

The `kalixlouiis/raw-data` corpus is a synthesized collection designed to represent the multifaceted nature of the Burmese language, covering formal, informal, and specialized domains. The dataset focuses on Unicode-standard text and incorporates code-mixed elements (Burmese-English) to reflect real-world language usage.

## Data Composition & Sources

The dataset is compiled from the following diverse repositories:

| Source Repository | Content Extracted |
| :--- | :--- |
| **DatarrX/PlanetFacts-QA-MY** | Educational "Question" and "Answer" pairs. |
| **DatarrX/burmese-VOA** | Contemporary news articles ("text"). |
| **DatarrX/mywikisource** | Historical and literary "context" data. |
| **DatarrX/tipitaka-dataset** | Classical religious and philosophical "content". |
| **kalixlouiis/Boyfriend-LLM-Instruct** | Conversational "assistant" responses from message lists. |
| **kalixlouiis/myanmar-linguistic-ambiguitie-001** | Verified "text" entries where `is_correct_in_context` equals 1. |
| **kalixlouiis/myanmar-written-spoken-text-pairs** | Natural "spoken_style" transcriptions. |
| **kalixlouiis/myanmar-wikipedia-corpus** | General knowledge and encyclopedic "text". |
| **kalixlouiis/Myanmar-English-translation** | Burmese ("my") segments from all splits (Train, Test, Val). |
| **kalixlouiis/general-burmese-sentences** | Wide-ranging "text" from general usage. |
| **kalixlouiis/myWattpad** | Modern creative writing and fictional "text". |

## Data Preprocessing & Quality Control

To ensure this dataset meets modern NLP standards, a rigorous cleaning pipeline was applied:

* **Encoding Standardization:** Removal of Zawgyi-encoded text and correction of Unicode rendering errors to ensure consistent character representation.
* **Noise Reduction:** Filtering of nonsensical sentences, corrupted strings, and non-linguistic artifacts.
* **Linguistic Validation:** Only contextually correct sentences were selected from linguistic ambiguity datasets to maintain grammatical standards.
* **Error Tolerance:** The dataset has a noise level estimated to be under 1-2%. While minimal noise may persist, it does not impact the overall quality for large-scale training.
* **Code-Mixed Handling:** Inclusion of natural code-mixed (Burmese + English) sentences to enhance model robustness in multilingual environments.

## Supported Tasks

* **Language Modeling:** Ideal for pre-training or fine-tuning causal language models.
* **Tokenizer Development:** High-quality data for training BPE, Unigram, or WordPiece tokenizers.
* **Feature Extraction:** Useful for generating embeddings for Burmese text.
* **Text Analysis:** Suitable for linguistic research and stylistic analysis of Burmese prose.

## Licensing Information

The dataset is distributed for open-source research and development. Please refer to the specific source repositories for individual data licensing constraints.