zehnlab/tajik-corpora-V2

--- dataset_info: features: - name: id dtype: string - name: dataset dtype: string - name: source_file dtype: string - name: source_url dtype: string - name: crawl_dump dtype: string - name: minhash_cluster_size dtype: int64 - name: language dtype: string - name: content dtype: string splits: - name: train num_bytes: 10386401225 num_examples: 1099814 - name: validation num_bytes: 83926870 num_examples: 8887 - name: dev num_bytes: 20993522 num_examples: 2223 download_size: 4995980299 dataset_size: 10491321617 configs: - config_name: default data_files: - split: train path: data/train-* - split: validation path: data/validation-* - split: dev path: data/dev-* --- Tajik Corpora V2 2.7B ## Total Tokens: 2.7B ## Datasets in Corpora | Dataset | Tokens (mln) | |---------|-------------| | FineWeb2 Tajik | 938.0 | | FineWEB EDU Translated to Tajik | 771.0 | | FinePDFs Tajik | 350.0 | | HPLT 3 Filtered Dedup | 341.0 | | General pretrain inhouse data: News, Books, Web | 298.0 |