chenzhe0000/wikitext_cleaned

---
license: mit
---

# Dataset Card for Cleaned & Chunked WikiText Corpus

## 📌 Dataset Description

This dataset is a cleaned and chunked version of WikiText-style corpus, designed for language model pretraining and evaluation.

The preprocessing pipeline follows a lightweight data curation paradigm:

> **cleaning → deduplication → normalization → token-aware chunking**

Specifically, the dataset includes:

- Removal of special and noisy characters
- Cleaning of formatting artifacts (e.g., HTML tags, irregular symbols)
- Deduplication to reduce redundant or highly similar text samples
- Text normalization (whitespace, encoding, etc.)
- Chunking of long documents into smaller segments suitable for model training

Each sample corresponds to a **cleaned text chunk**, rather than a full original document.

---

## 📊 Data Structure

Each sample in the dataset follows this structure:

```json
{
"uid": "string",
"content": "string",
"meta_data": {
"index": "int",
"total": "int",
"length": "int"
}
}