Ba2han/tokenized-20-11
收藏Hugging Face2025-11-20 更新2025-12-20 收录
下载链接：
https://hf-mirror.com/datasets/Ba2han/tokenized-20-11
下载链接
链接失效反馈官方服务：
资源简介：
---
license: mit
language:
- tr
- en
tags:
- tokenized
- qwen
size_categories:
- 10B<n<100B
---
# Tokenized Dataset: Ba2han/tokenized-20-11
This dataset contains pre-tokenized `input_ids` optimized for training.
## Processing Details
- **Tokenizer**: `unsloth/qwen3-0.6b`
- **Sequence Length**: `50` < tokens < `4000`
- **Deduplication**: Exact string deduplication applied across all sources.
- **Format**: Parquet files with a single `input_ids` column. `EOS` token added to every example.
## Source Datasets
The following datasets were streamed, filtered, and combined:
1. **Turkish Corpora (Ba2han Collection)**
- `Ba2han/vngrs-web-filtered`
- `Ba2han/finepdfs-filtered-edu-tr`
- `Ba2han/fineweb2-filtered-edu-tr`
- `Ba2han/c4-fineweb-filtered`
- `Ba2han/synth-tr`
2. **PleIAs/SYNTH**
- **Files**: `synth_30.parquet` to `synth_60.parquet`
- **Filters**:
- Language = 'en'
- Word count < 950
- Constraints is Empty, NaN, or None.
- **Format**: `synthetic_reasoning + \n + synthetic_answer`
3. **facebook/principia-collection**
- **Split**: `numerical`
- **Format**: `problem_statement + \n + answer`
提供机构：
Ba2han
