wikimedia/wikisource

Name: wikimedia/wikisource
Creator: wikimedia
Published: 2023-12-08 13:36:41
License: 暂无描述

Hugging Face2023-12-08 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/wikimedia/wikisource

下载链接

链接失效反馈

官方服务：

资源简介：

--- language: - ar - as - az - ban - be - bg - bn - br - bs - ca - cs - cy - da - de - el - en - eo - es - et - eu - fa - fi - fo - fr - gl - gu - he - hi - hr - hu - hy - id - is - it - ja - jv - kn - ko - la - li - lij - lt - mk - ml - mr - nan - nap - nl - 'no' - or - pa - pl - pms - pt - ro - ru - sa - sah - sk - sl - sr - su - sv - ta - te - th - tr - uk - vec - vi - wa - yi - zh license: - cc-by-sa-3.0 - gfdl size_categories: - n<1K - 1K<n<10K - 10K<n<100K - 100K<n<1M task_categories: - text-generation - fill-mask task_ids: - language-modeling - masked-language-modeling dataset_info: - config_name: 20231201.ar features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1027384499 num_examples: 38235 download_size: 471633595 dataset_size: 1027384499 - config_name: 20231201.as features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 10334689 num_examples: 1191 download_size: 3976908 dataset_size: 10334689 - config_name: 20231201.az features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 37612618 num_examples: 9706 download_size: 20953203 dataset_size: 37612618 - config_name: 20231201.ban features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 468189 num_examples: 591 download_size: 169732 dataset_size: 468189 - config_name: 20231201.be features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 52555230 num_examples: 4876 download_size: 26356864 dataset_size: 52555230 - config_name: 20231201.bg features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 33320786 num_examples: 2316 download_size: 14416495 dataset_size: 33320786 - config_name: 20231201.bn features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 12256 num_examples: 5 download_size: 11958 dataset_size: 12256 - config_name: 20231201.br features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 179457 num_examples: 314 download_size: 89388 dataset_size: 179457 - config_name: 20231201.bs features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 15735639 num_examples: 1918 download_size: 9427044 dataset_size: 15735639 - config_name: 20231201.ca features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 9470138 num_examples: 1229 download_size: 5021947 dataset_size: 9470138 - config_name: 20231201.cs features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 190358421 num_examples: 42735 download_size: 124249346 dataset_size: 190358421 - config_name: 20231201.cy features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2161046 num_examples: 1090 download_size: 1251259 dataset_size: 2161046 - config_name: 20231201.da features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 18564343 num_examples: 1043 download_size: 10957998 dataset_size: 18564343 - config_name: 20231201.de features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 527146739 num_examples: 141657 download_size: 312816088 dataset_size: 527146739 - config_name: 20231201.el features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 215554120 num_examples: 8024 download_size: 103217935 dataset_size: 215554120 - config_name: 20231201.en features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2470274084 num_examples: 208279 download_size: 1382960909 dataset_size: 2470274084 - config_name: 20231201.eo features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3207308 num_examples: 384 download_size: 2009128 dataset_size: 3207308 - config_name: 20231201.es features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 381152287 num_examples: 37831 download_size: 224097690 dataset_size: 381152287 - config_name: 20231201.et features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3906488 num_examples: 722 download_size: 2316406 dataset_size: 3906488 - config_name: 20231201.eu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 17014224 num_examples: 923 download_size: 9473130 dataset_size: 17014224 - config_name: 20231201.fa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 79812303 num_examples: 5751 download_size: 33916994 dataset_size: 79812303 - config_name: 20231201.fi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 55271379 num_examples: 13414 download_size: 33265827 dataset_size: 55271379 - config_name: 20231201.fo features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 174113 num_examples: 62 download_size: 112092 dataset_size: 174113 - config_name: 20231201.fr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 90126375 num_examples: 23201 download_size: 49429480 dataset_size: 90126375 - config_name: 20231201.gl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6679826 num_examples: 747 download_size: 3712275 dataset_size: 6679826 - config_name: 20231201.gu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2430315 num_examples: 797 download_size: 948872 dataset_size: 2430315 - config_name: 20231201.he features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1166312812 num_examples: 107248 download_size: 519792862 dataset_size: 1166312812 - config_name: 20231201.hi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2203936 num_examples: 3494 download_size: 443194 dataset_size: 2203936 - config_name: 20231201.hr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 61069921 num_examples: 8278 download_size: 38797697 dataset_size: 61069921 - config_name: 20231201.hu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 94429364 num_examples: 20846 download_size: 62012894 dataset_size: 94429364 - config_name: 20231201.hy features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 39941751 num_examples: 2248 download_size: 18574182 dataset_size: 39941751 - config_name: 20231201.id features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 40100527 num_examples: 2234 download_size: 18175030 dataset_size: 40100527 - config_name: 20231201.is features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 20657687 num_examples: 4880 download_size: 11620112 dataset_size: 20657687 - config_name: 20231201.it features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 211472789 num_examples: 65047 download_size: 115227856 dataset_size: 211472789 - config_name: 20231201.ja features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 202476246 num_examples: 11879 download_size: 90838204 dataset_size: 202476246 - config_name: 20231201.jv features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6977954 num_examples: 534 download_size: 3409151 dataset_size: 6977954 - config_name: 20231201.kn features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 202914626 num_examples: 14980 download_size: 73290389 dataset_size: 202914626 - config_name: 20231201.ko features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 661997703 num_examples: 24858 download_size: 302950424 dataset_size: 661997703 - config_name: 20231201.la features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 583348181 num_examples: 11032 download_size: 351767028 dataset_size: 583348181 - config_name: 20231201.li features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2143869 num_examples: 1857 download_size: 1191398 dataset_size: 2143869 - config_name: 20231201.lij features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1281480 num_examples: 1185 download_size: 651083 dataset_size: 1281480 - config_name: 20231201.lt features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 7513991 num_examples: 1874 download_size: 4637316 dataset_size: 7513991 - config_name: 20231201.mk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 12706090 num_examples: 2166 download_size: 5077478 dataset_size: 12706090 - config_name: 20231201.ml features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 81611041 num_examples: 6052 download_size: 29462281 dataset_size: 81611041 - config_name: 20231201.mr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 35302346 num_examples: 1485 download_size: 13300483 dataset_size: 35302346 - config_name: 20231201.nap features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 91852 num_examples: 155 download_size: 53478 dataset_size: 91852 - config_name: 20231201.nl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 48325965 num_examples: 5260 download_size: 27915130 dataset_size: 48325965 - config_name: 20231201.no features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2289098 num_examples: 379 download_size: 1397633 dataset_size: 2289098 - config_name: 20231201.or features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 18535382 num_examples: 693 download_size: 7348706 dataset_size: 18535382 - config_name: 20231201.pa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6536266 num_examples: 107 download_size: 2583902 dataset_size: 6536266 - config_name: 20231201.pl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 56457491 num_examples: 12020 download_size: 34312764 dataset_size: 56457491 - config_name: 20231201.pms features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 16256157 num_examples: 4093 download_size: 9703819 dataset_size: 16256157 - config_name: 20231201.pt features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 106619107 num_examples: 23171 download_size: 62791422 dataset_size: 106619107 - config_name: 20231201.ro features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 134273629 num_examples: 12921 download_size: 81375524 dataset_size: 134273629 - config_name: 20231201.ru features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 9393299725 num_examples: 372768 download_size: 4601162148 dataset_size: 9393299725 - config_name: 20231201.sa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 636225055 num_examples: 22986 download_size: 231955608 dataset_size: 636225055 - config_name: 20231201.sah features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 17305188 num_examples: 903 download_size: 7654932 dataset_size: 17305188 - config_name: 20231201.sk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3532173 num_examples: 390 download_size: 2217851 dataset_size: 3532173 - config_name: 20231201.sl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 366151991 num_examples: 17267 download_size: 242655257 dataset_size: 366151991 - config_name: 20231201.sr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 195710992 num_examples: 38987 download_size: 86833442 dataset_size: 195710992 - config_name: 20231201.su features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 356902 num_examples: 20 download_size: 220452 dataset_size: 356902 - config_name: 20231201.sv features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 27912113 num_examples: 6296 download_size: 16513469 dataset_size: 27912113 - config_name: 20231201.ta features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 113836204 num_examples: 4702 download_size: 40070603 dataset_size: 113836204 - config_name: 20231201.te features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 94840451 num_examples: 9012 download_size: 36668092 dataset_size: 94840451 - config_name: 20231201.th features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 73437990 num_examples: 2383 download_size: 23644914 dataset_size: 73437990 - config_name: 20231201.tr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 64957772 num_examples: 7220 download_size: 34039502 dataset_size: 64957772 - config_name: 20231201.uk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 46059083 num_examples: 4171 download_size: 21135029 dataset_size: 46059083 - config_name: 20231201.vec features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 5700371 num_examples: 3492 download_size: 3097037 dataset_size: 5700371 - config_name: 20231201.vi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 48099940 num_examples: 5471 download_size: 17336608 dataset_size: 48099940 - config_name: 20231201.wa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3735624 num_examples: 897 download_size: 2222694 dataset_size: 3735624 - config_name: 20231201.yi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 24802558 num_examples: 1669 download_size: 10686751 dataset_size: 24802558 - config_name: 20231201.zh features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 5070438246 num_examples: 265669 download_size: 3309500049 dataset_size: 5070438246 - config_name: 20231201.zh-min-nan features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 21109492 num_examples: 2360 download_size: 10288524 dataset_size: 21109492 configs: - config_name: 20231201.ar data_files: - split: train path: 20231201.ar/train-* - config_name: 20231201.as data_files: - split: train path: 20231201.as/train-* - config_name: 20231201.az data_files: - split: train path: 20231201.az/train-* - config_name: 20231201.ban data_files: - split: train path: 20231201.ban/train-* - config_name: 20231201.be data_files: - split: train path: 20231201.be/train-* - config_name: 20231201.bg data_files: - split: train path: 20231201.bg/train-* - config_name: 20231201.bn data_files: - split: train path: 20231201.bn/train-* - config_name: 20231201.br data_files: - split: train path: 20231201.br/train-* - config_name: 20231201.bs data_files: - split: train path: 20231201.bs/train-* - config_name: 20231201.ca data_files: - split: train path: 20231201.ca/train-* - config_name: 20231201.cs data_files: - split: train path: 20231201.cs/train-* - config_name: 20231201.cy data_files: - split: train path: 20231201.cy/train-* - config_name: 20231201.da data_files: - split: train path: 20231201.da/train-* - config_name: 20231201.de data_files: - split: train path: 20231201.de/train-* - config_name: 20231201.el data_files: - split: train path: 20231201.el/train-* - config_name: 20231201.en data_files: - split: train path: 20231201.en/train-* - config_name: 20231201.eo data_files: - split: train path: 20231201.eo/train-* - config_name: 20231201.es data_files: - split: train path: 20231201.es/train-* - config_name: 20231201.et data_files: - split: train path: 20231201.et/train-* - config_name: 20231201.eu data_files: - split: train path: 20231201.eu/train-* - config_name: 20231201.fa data_files: - split: train path: 20231201.fa/train-* - config_name: 20231201.fi data_files: - split: train path: 20231201.fi/train-* - config_name: 20231201.fo data_files: - split: train path: 20231201.fo/train-* - config_name: 20231201.fr data_files: - split: train path: 20231201.fr/train-* - config_name: 20231201.gl data_files: - split: train path: 20231201.gl/train-* - config_name: 20231201.gu data_files: - split: train path: 20231201.gu/train-* - config_name: 20231201.he data_files: - split: train path: 20231201.he/train-* - config_name: 20231201.hi data_files: - split: train path: 20231201.hi/train-* - config_name: 20231201.hr data_files: - split: train path: 20231201.hr/train-* - config_name: 20231201.hu data_files: - split: train path: 20231201.hu/train-* - config_name: 20231201.hy data_files: - split: train path: 20231201.hy/train-* - config_name: 20231201.id data_files: - split: train path: 20231201.id/train-* - config_name: 20231201.is data_files: - split: train path: 20231201.is/train-* - config_name: 20231201.it data_files: - split: train path: 20231201.it/train-* - config_name: 20231201.ja data_files: - split: train path: 20231201.ja/train-* - config_name: 20231201.jv data_files: - split: train path: 20231201.jv/train-* - config_name: 20231201.kn data_files: - split: train path: 20231201.kn/train-* - config_name: 20231201.ko data_files: - split: train path: 20231201.ko/train-* - config_name: 20231201.la data_files: - split: train path: 20231201.la/train-* - config_name: 20231201.li data_files: - split: train path: 20231201.li/train-* - config_name: 20231201.lij data_files: - split: train path: 20231201.lij/train-* - config_name: 20231201.lt data_files: - split: train path: 20231201.lt/train-* - config_name: 20231201.mk data_files: - split: train path: 20231201.mk/train-* - config_name: 20231201.ml data_files: - split: train path: 20231201.ml/train-* - config_name: 20231201.mr data_files: - split: train path: 20231201.mr/train-* - config_name: 20231201.nap data_files: - split: train path: 20231201.nap/train-* - config_name: 20231201.nl data_files: - split: train path: 20231201.nl/train-* - config_name: 20231201.no data_files: - split: train path: 20231201.no/train-* - config_name: 20231201.or data_files: - split: train path: 20231201.or/train-* - config_name: 20231201.pa data_files: - split: train path: 20231201.pa/train-* - config_name: 20231201.pl data_files: - split: train path: 20231201.pl/train-* - config_name: 20231201.pms data_files: - split: train path: 20231201.pms/train-* - config_name: 20231201.pt data_files: - split: train path: 20231201.pt/train-* - config_name: 20231201.ro data_files: - split: train path: 20231201.ro/train-* - config_name: 20231201.ru data_files: - split: train path: 20231201.ru/train-* - config_name: 20231201.sa data_files: - split: train path: 20231201.sa/train-* - config_name: 20231201.sah data_files: - split: train path: 20231201.sah/train-* - config_name: 20231201.sk data_files: - split: train path: 20231201.sk/train-* - config_name: 20231201.sl data_files: - split: train path: 20231201.sl/train-* - config_name: 20231201.sr data_files: - split: train path: 20231201.sr/train-* - config_name: 20231201.su data_files: - split: train path: 20231201.su/train-* - config_name: 20231201.sv data_files: - split: train path: 20231201.sv/train-* - config_name: 20231201.ta data_files: - split: train path: 20231201.ta/train-* - config_name: 20231201.te data_files: - split: train path: 20231201.te/train-* - config_name: 20231201.th data_files: - split: train path: 20231201.th/train-* - config_name: 20231201.tr data_files: - split: train path: 20231201.tr/train-* - config_name: 20231201.uk data_files: - split: train path: 20231201.uk/train-* - config_name: 20231201.vec data_files: - split: train path: 20231201.vec/train-* - config_name: 20231201.vi data_files: - split: train path: 20231201.vi/train-* - config_name: 20231201.wa data_files: - split: train path: 20231201.wa/train-* - config_name: 20231201.yi data_files: - split: train path: 20231201.yi/train-* - config_name: 20231201.zh data_files: - split: train path: 20231201.zh/train-* - config_name: 20231201.zh-min-nan data_files: - split: train path: 20231201.zh-min-nan/train-* --- # Dataset Card for Wikimedia Wikisource ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** https://dumps.wikimedia.org - **Repository:** - **Paper:** - **Leaderboard:** - **Point of Contact:** ### Dataset Summary Wikisource dataset containing cleaned articles of all languages. The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikisource text with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump, and you can load them by date and language like this: ```python from datasets import load_dataset ds = load_dataset("wikimedia/wikisource", "20231201.en") ``` ### Supported Tasks and Leaderboards The dataset is generally used for Language Modeling. ### Languages You can find the list of all languages here: https://meta.wikimedia.org/wiki/Wikisource#List_of_Wikisources Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual Wikisource here: https://wikisource.org/wiki/Wikisource:Languages ## Dataset Structure ### Data Instances An example looks as follows: ``` {'id': '36', 'url': 'https://ca.wikisource.org/wiki/Comunicat%20de%20Berl%C3%ADn', 'title': 'Comunicat de Berlín', 'text': "\n\nPreàmbul \nEl 19 de juny de 1999, un any després de la Declaració de la Sorbona,..." } ``` ### Data Fields The data fields are the same among all language configurations: - `id` (`str`): ID of the text. - `url` (`str`): URL of the text. - `title` (`str`): Title of the text. - `text` (`str`): Content of the text. ### Data Splits All language configurations contain a single `train` split. ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization The dataset is built from the Wikisource dumps: https://dumps.wikimedia.org You can find the full list of languages and dates here: https://dumps.wikimedia.org/backup-index.html The articles have been parsed using the [`mwparserfromhell`](https://mwparserfromhell.readthedocs.io) tool. #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information Copyright licensing information: https://dumps.wikimedia.org/legal.html All original textual content is licensed under the [GNU Free Documentation License](https://www.gnu.org/licenses/fdl-1.3.html) (GFDL) and the [Creative Commons Attribution-Share-Alike 3.0 License](https://creativecommons.org/licenses/by-sa/3.0/). Some text may be available only under the Creative Commons license; see their [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) for details. Text written by some authors may be released under additional licenses or into the public domain. ### Citation Information ``` @ONLINE{wikidump, author = "Wikimedia Foundation", title = "Wikimedia Downloads", url = "https://dumps.wikimedia.org" } ``` ### Contributions Thanks to [@albertvillanova](https://huggingface.co/albertvillanova) for adding this dataset.

提供机构：

wikimedia

原始信息汇总

数据集概述

语言支持

支持多种语言，包括但不限于阿拉伯语（ar）、阿萨姆语（as）、阿塞拜疆语（az）等。

许可证

数据集遵循的许可证包括：cc-by-sa-3.0 和 gfdl。

大小分类

数据集根据大小分为四个类别：n<1K, 1K<n<10K, 10K<n<100K, 100K<n<1M。

任务分类

主要任务类别包括：text-generation 和 fill-mask。
具体任务ID包括：language-modeling 和 masked-language-modeling。

数据集详细信息

配置名称: 20231201.xx（xx代表不同语言代码）
特征:
- id: 数据集中的唯一标识符，数据类型为string。
- url: 数据集中的链接地址，数据类型为string。
- title: 数据集中的标题，数据类型为string。
- text: 数据集中的文本内容，数据类型为string。
分割:
- train: 训练集，包含具体的数据量（num_bytes）和示例数量（num_examples）。
- 示例数据量（num_bytes）和下载大小（download_size）以及数据集大小（dataset_size）均详细列出。

示例数据量统计

以20231201.ar为例，训练集数据量为1027384499字节，包含38235个示例，下载大小为471633595字节，数据集总大小为1027384499字节。
其他语言的数据量统计类似，均详细列出了训练集的数据量和示例数量。

5,000+

优质数据集

54 个

任务类型

进入经典数据集