Ba2han/mixed_corpus-2303

---
dataset_info:
features:
- name: input_ids
sequence: uint16
---
# Mixed Corpus 2303

This dataset is a shuffled combination of the following tokenized datasets.
Only the `input_ids` column has been retained, and the values have been cast to `uint16` to save space.

## Token Amounts

| Source Dataset | Token Count |
| :--- | :--- |
| [Ba2han/forumsoh_tokenized](https://huggingface.co/datasets/Ba2han/forumsoh_tokenized) | 4,232,493,403 |
| [Ba2han/3.5b-tokens](https://huggingface.co/datasets/Ba2han/3.5b-tokens) | 2,879,132,572 |
| [Ba2han/long-corpus](https://huggingface.co/datasets/Ba2han/long-corpus) | 6,072,626,857 |
| [Ba2han/English_mix-tokenized](https://huggingface.co/datasets/Ba2han/English_mix-tokenized) | 1,027,372,236 |
| **Total** | **14,211,625,068** |