wikimedia/wikisource

Name: wikimedia/wikisource
Creator: wikimedia
Published: 2023-12-08 13:36:41
License: 暂无描述

Hugging Face2023-12-08 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/wikimedia/wikisource

下载链接

链接失效反馈

官方服务：

资源简介：

--- language: - ar - as - az - ban - be - bg - bn - br - bs - ca - cs - cy - da - de - el - en - eo - es - et - eu - fa - fi - fo - fr - gl - gu - he - hi - hr - hu - hy - id - is - it - ja - jv - kn - ko - la - li - lij - lt - mk - ml - mr - nan - nap - nl - 'no' - or - pa - pl - pms - pt - ro - ru - sa - sah - sk - sl - sr - su - sv - ta - te - th - tr - uk - vec - vi - wa - yi - zh license: - cc-by-sa-3.0 - gfdl size_categories: - n<1K - 1K<n<10K - 10K<n<100K - 100K<n<1M task_categories: - text-generation - fill-mask task_ids: - language-modeling - masked-language-modeling dataset_info: - config_name: 20231201.ar features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1027384499 num_examples: 38235 download_size: 471633595 dataset_size: 1027384499 - config_name: 20231201.as features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 10334689 num_examples: 1191 download_size: 3976908 dataset_size: 10334689 - config_name: 20231201.az features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 37612618 num_examples: 9706 download_size: 20953203 dataset_size: 37612618 - config_name: 20231201.ban features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 468189 num_examples: 591 download_size: 169732 dataset_size: 468189 - config_name: 20231201.be features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 52555230 num_examples: 4876 download_size: 26356864 dataset_size: 52555230 - config_name: 20231201.bg features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 33320786 num_examples: 2316 download_size: 14416495 dataset_size: 33320786 - config_name: 20231201.bn features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 12256 num_examples: 5 download_size: 11958 dataset_size: 12256 - config_name: 20231201.br features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 179457 num_examples: 314 download_size: 89388 dataset_size: 179457 - config_name: 20231201.bs features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 15735639 num_examples: 1918 download_size: 9427044 dataset_size: 15735639 - config_name: 20231201.ca features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 9470138 num_examples: 1229 download_size: 5021947 dataset_size: 9470138 - config_name: 20231201.cs features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 190358421 num_examples: 42735 download_size: 124249346 dataset_size: 190358421 - config_name: 20231201.cy features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2161046 num_examples: 1090 download_size: 1251259 dataset_size: 2161046 - config_name: 20231201.da features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 18564343 num_examples: 1043 download_size: 10957998 dataset_size: 18564343 - config_name: 20231201.de features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 527146739 num_examples: 141657 download_size: 312816088 dataset_size: 527146739 - config_name: 20231201.el features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 215554120 num_examples: 8024 download_size: 103217935 dataset_size: 215554120 - config_name: 20231201.en features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2470274084 num_examples: 208279 download_size: 1382960909 dataset_size: 2470274084 - config_name: 20231201.eo features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3207308 num_examples: 384 download_size: 2009128 dataset_size: 3207308 - config_name: 20231201.es features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 381152287 num_examples: 37831 download_size: 224097690 dataset_size: 381152287 - config_name: 20231201.et features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3906488 num_examples: 722 download_size: 2316406 dataset_size: 3906488 - config_name: 20231201.eu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 17014224 num_examples: 923 download_size: 9473130 dataset_size: 17014224 - config_name: 20231201.fa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 79812303 num_examples: 5751 download_size: 33916994 dataset_size: 79812303 - config_name: 20231201.fi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 55271379 num_examples: 13414 download_size: 33265827 dataset_size: 55271379 - config_name: 20231201.fo features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 174113 num_examples: 62 download_size: 112092 dataset_size: 174113 - config_name: 20231201.fr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 90126375 num_examples: 23201 download_size: 49429480 dataset_size: 90126375 - config_name: 20231201.gl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6679826 num_examples: 747 download_size: 3712275 dataset_size: 6679826 - config_name: 20231201.gu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2430315 num_examples: 797 download_size: 948872 dataset_size: 2430315 - config_name: 20231201.he features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1166312812 num_examples: 107248 download_size: 519792862 dataset_size: 1166312812 - config_name: 20231201.hi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2203936 num_examples: 3494 download_size: 443194 dataset_size: 2203936 - config_name: 20231201.hr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 61069921 num_examples: 8278 download_size: 38797697 dataset_size: 61069921 - config_name: 20231201.hu features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 94429364 num_examples: 20846 download_size: 62012894 dataset_size: 94429364 - config_name: 20231201.hy features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 39941751 num_examples: 2248 download_size: 18574182 dataset_size: 39941751 - config_name: 20231201.id features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 40100527 num_examples: 2234 download_size: 18175030 dataset_size: 40100527 - config_name: 20231201.is features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 20657687 num_examples: 4880 download_size: 11620112 dataset_size: 20657687 - config_name: 20231201.it features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 211472789 num_examples: 65047 download_size: 115227856 dataset_size: 211472789 - config_name: 20231201.ja features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 202476246 num_examples: 11879 download_size: 90838204 dataset_size: 202476246 - config_name: 20231201.jv features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6977954 num_examples: 534 download_size: 3409151 dataset_size: 6977954 - config_name: 20231201.kn features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 202914626 num_examples: 14980 download_size: 73290389 dataset_size: 202914626 - config_name: 20231201.ko features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 661997703 num_examples: 24858 download_size: 302950424 dataset_size: 661997703 - config_name: 20231201.la features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 583348181 num_examples: 11032 download_size: 351767028 dataset_size: 583348181 - config_name: 20231201.li features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2143869 num_examples: 1857 download_size: 1191398 dataset_size: 2143869 - config_name: 20231201.lij features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 1281480 num_examples: 1185 download_size: 651083 dataset_size: 1281480 - config_name: 20231201.lt features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 7513991 num_examples: 1874 download_size: 4637316 dataset_size: 7513991 - config_name: 20231201.mk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 12706090 num_examples: 2166 download_size: 5077478 dataset_size: 12706090 - config_name: 20231201.ml features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 81611041 num_examples: 6052 download_size: 29462281 dataset_size: 81611041 - config_name: 20231201.mr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 35302346 num_examples: 1485 download_size: 13300483 dataset_size: 35302346 - config_name: 20231201.nap features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 91852 num_examples: 155 download_size: 53478 dataset_size: 91852 - config_name: 20231201.nl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 48325965 num_examples: 5260 download_size: 27915130 dataset_size: 48325965 - config_name: 20231201.no features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 2289098 num_examples: 379 download_size: 1397633 dataset_size: 2289098 - config_name: 20231201.or features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 18535382 num_examples: 693 download_size: 7348706 dataset_size: 18535382 - config_name: 20231201.pa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 6536266 num_examples: 107 download_size: 2583902 dataset_size: 6536266 - config_name: 20231201.pl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 56457491 num_examples: 12020 download_size: 34312764 dataset_size: 56457491 - config_name: 20231201.pms features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 16256157 num_examples: 4093 download_size: 9703819 dataset_size: 16256157 - config_name: 20231201.pt features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 106619107 num_examples: 23171 download_size: 62791422 dataset_size: 106619107 - config_name: 20231201.ro features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 134273629 num_examples: 12921 download_size: 81375524 dataset_size: 134273629 - config_name: 20231201.ru features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 9393299725 num_examples: 372768 download_size: 4601162148 dataset_size: 9393299725 - config_name: 20231201.sa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 636225055 num_examples: 22986 download_size: 231955608 dataset_size: 636225055 - config_name: 20231201.sah features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 17305188 num_examples: 903 download_size: 7654932 dataset_size: 17305188 - config_name: 20231201.sk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3532173 num_examples: 390 download_size: 2217851 dataset_size: 3532173 - config_name: 20231201.sl features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 366151991 num_examples: 17267 download_size: 242655257 dataset_size: 366151991 - config_name: 20231201.sr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 195710992 num_examples: 38987 download_size: 86833442 dataset_size: 195710992 - config_name: 20231201.su features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 356902 num_examples: 20 download_size: 220452 dataset_size: 356902 - config_name: 20231201.sv features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 27912113 num_examples: 6296 download_size: 16513469 dataset_size: 27912113 - config_name: 20231201.ta features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 113836204 num_examples: 4702 download_size: 40070603 dataset_size: 113836204 - config_name: 20231201.te features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 94840451 num_examples: 9012 download_size: 36668092 dataset_size: 94840451 - config_name: 20231201.th features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 73437990 num_examples: 2383 download_size: 23644914 dataset_size: 73437990 - config_name: 20231201.tr features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 64957772 num_examples: 7220 download_size: 34039502 dataset_size: 64957772 - config_name: 20231201.uk features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 46059083 num_examples: 4171 download_size: 21135029 dataset_size: 46059083 - config_name: 20231201.vec features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 5700371 num_examples: 3492 download_size: 3097037 dataset_size: 5700371 - config_name: 20231201.vi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 48099940 num_examples: 5471 download_size: 17336608 dataset_size: 48099940 - config_name: 20231201.wa features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 3735624 num_examples: 897 download_size: 2222694 dataset_size: 3735624 - config_name: 20231201.yi features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 24802558 num_examples: 1669 download_size: 10686751 dataset_size: 24802558 - config_name: 20231201.zh features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 5070438246 num_examples: 265669 download_size: 3309500049 dataset_size: 5070438246 - config_name: 20231201.zh-min-nan features: - name: id dtype: string - name: url dtype: string - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 21109492 num_examples: 2360 download_size: 10288524 dataset_size: 21109492 configs: - config_name: 20231201.ar data_files: - split: train path: 20231201.ar/train-* - config_name: 20231201.as data_files: - split: train path: 20231201.as/train-* - config_name: 20231201.az data_files: - split: train path: 20231201.az/train-* - config_name: 20231201.ban data_files: - split: train path: 20231201.ban/train-* - config_name: 20231201.be data_files: - split: train path: 20231201.be/train-* - config_name: 20231201.bg data_files: - split: train path: 20231201.bg/train-* - config_name: 20231201.bn data_files: - split: train path: 20231201.bn/train-* - config_name: 20231201.br data_files: - split: train path: 20231201.br/train-* - config_name: 20231201.bs data_files: - split: train path: 20231201.bs/train-* - config_name: 20231201.ca data_files: - split: train path: 20231201.ca/train-* - config_name: 20231201.cs data_files: - split: train path: 20231201.cs/train-* - config_name: 20231201.cy data_files: - split: train path: 20231201.cy/train-* - config_name: 20231201.da data_files: - split: train path: 20231201.da/train-* - config_name: 20231201.de data_files: - split: train path: 20231201.de/train-* - config_name: 20231201.el data_files: - split: train path: 20231201.el/train-* - config_name: 20231201.en data_files: - split: train path: 20231201.en/train-* - config_name: 20231201.eo data_files: - split: train path: 20231201.eo/train-* - config_name: 20231201.es data_files: - split: train path: 20231201.es/train-* - config_name: 20231201.et data_files: - split: train path: 20231201.et/train-* - config_name: 20231201.eu data_files: - split: train path: 20231201.eu/train-* - config_name: 20231201.fa data_files: - split: train path: 20231201.fa/train-* - config_name: 20231201.fi data_files: - split: train path: 20231201.fi/train-* - config_name: 20231201.fo data_files: - split: train path: 20231201.fo/train-* - config_name: 20231201.fr data_files: - split: train path: 20231201.fr/train-* - config_name: 20231201.gl data_files: - split: train path: 20231201.gl/train-* - config_name: 20231201.gu data_files: - split: train path: 20231201.gu/train-* - config_name: 20231201.he data_files: - split: train path: 20231201.he/train-* - config_name: 20231201.hi data_files: - split: train path: 20231201.hi/train-* - config_name: 20231201.hr data_files: - split: train path: 20231201.hr/train-* - config_name: 20231201.hu data_files: - split: train path: 20231201.hu/train-* - config_name: 20231201.hy data_files: - split: train path: 20231201.hy/train-* - config_name: 20231201.id data_files: - split: train path: 20231201.id/train-* - config_name: 20231201.is data_files: - split: train path: 20231201.is/train-* - config_name: 20231201.it data_files: - split: train path: 20231201.it/train-* - config_name: 20231201.ja data_files: - split: train path: 20231201.ja/train-* - config_name: 20231201.jv data_files: - split: train path: 20231201.jv/train-* - config_name: 20231201.kn data_files: - split: train path: 20231201.kn/train-* - config_name: 20231201.ko data_files: - split: train path: 20231201.ko/train-* - config_name: 20231201.la data_files: - split: train path: 20231201.la/train-* - config_name: 20231201.li data_files: - split: train path: 20231201.li/train-* - config_name: 20231201.lij data_files: - split: train path: 20231201.lij/train-* - config_name: 20231201.lt data_files: - split: train path: 20231201.lt/train-* - config_name: 20231201.mk data_files: - split: train path: 20231201.mk/train-* - config_name: 20231201.ml data_files: - split: train path: 20231201.ml/train-* - config_name: 20231201.mr data_files: - split: train path: 20231201.mr/train-* - config_name: 20231201.nap data_files: - split: train path: 20231201.nap/train-* - config_name: 20231201.nl data_files: - split: train path: 20231201.nl/train-* - config_name: 20231201.no data_files: - split: train path: 20231201.no/train-* - config_name: 20231201.or data_files: - split: train path: 20231201.or/train-* - config_name: 20231201.pa data_files: - split: train path: 20231201.pa/train-* - config_name: 20231201.pl data_files: - split: train path: 20231201.pl/train-* - config_name: 20231201.pms data_files: - split: train path: 20231201.pms/train-* - config_name: 20231201.pt data_files: - split: train path: 20231201.pt/train-* - config_name: 20231201.ro data_files: - split: train path: 20231201.ro/train-* - config_name: 20231201.ru data_files: - split: train path: 20231201.ru/train-* - config_name: 20231201.sa data_files: - split: train path: 20231201.sa/train-* - config_name: 20231201.sah data_files: - split: train path: 20231201.sah/train-* - config_name: 20231201.sk data_files: - split: train path: 20231201.sk/train-* - config_name: 20231201.sl data_files: - split: train path: 20231201.sl/train-* - config_name: 20231201.sr data_files: - split: train path: 20231201.sr/train-* - config_name: 20231201.su data_files: - split: train path: 20231201.su/train-* - config_name: 20231201.sv data_files: - split: train path: 20231201.sv/train-* - config_name: 20231201.ta data_files: - split: train path: 20231201.ta/train-* - config_name: 20231201.te data_files: - split: train path: 20231201.te/train-* - config_name: 20231201.th data_files: - split: train path: 20231201.th/train-* - config_name: 20231201.tr data_files: - split: train path: 20231201.tr/train-* - config_name: 20231201.uk data_files: - split: train path: 20231201.uk/train-* - config_name: 20231201.vec data_files: - split: train path: 20231201.vec/train-* - config_name: 20231201.vi data_files: - split: train path: 20231201.vi/train-* - config_name: 20231201.wa data_files: - split: train path: 20231201.wa/train-* - config_name: 20231201.yi data_files: - split: train path: 20231201.yi/train-* - config_name: 20231201.zh data_files: - split: train path: 20231201.zh/train-* - config_name: 20231201.zh-min-nan data_files: - split: train path: 20231201.zh-min-nan/train-* --- # Dataset Card for Wikimedia Wikisource ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** https://dumps.wikimedia.org - **Repository:** - **Paper:** - **Leaderboard:** - **Point of Contact:** ### Dataset Summary Wikisource dataset containing cleaned articles of all languages. The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikisource text with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump, and you can load them by date and language like this: ```python from datasets import load_dataset ds = load_dataset("wikimedia/wikisource", "20231201.en") ``` ### Supported Tasks and Leaderboards The dataset is generally used for Language Modeling. ### Languages You can find the list of all languages here: https://meta.wikimedia.org/wiki/Wikisource#List_of_Wikisources Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual Wikisource here: https://wikisource.org/wiki/Wikisource:Languages ## Dataset Structure ### Data Instances An example looks as follows: ``` {'id': '36', 'url': 'https://ca.wikisource.org/wiki/Comunicat%20de%20Berl%C3%ADn', 'title': 'Comunicat de Berlín', 'text': "\n\nPreàmbul \nEl 19 de juny de 1999, un any després de la Declaració de la Sorbona,..." } ``` ### Data Fields The data fields are the same among all language configurations: - `id` (`str`): ID of the text. - `url` (`str`): URL of the text. - `title` (`str`): Title of the text. - `text` (`str`): Content of the text. ### Data Splits All language configurations contain a single `train` split. ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization The dataset is built from the Wikisource dumps: https://dumps.wikimedia.org You can find the full list of languages and dates here: https://dumps.wikimedia.org/backup-index.html The articles have been parsed using the [`mwparserfromhell`](https://mwparserfromhell.readthedocs.io) tool. #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information Copyright licensing information: https://dumps.wikimedia.org/legal.html All original textual content is licensed under the [GNU Free Documentation License](https://www.gnu.org/licenses/fdl-1.3.html) (GFDL) and the [Creative Commons Attribution-Share-Alike 3.0 License](https://creativecommons.org/licenses/by-sa/3.0/). Some text may be available only under the Creative Commons license; see their [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) for details. Text written by some authors may be released under additional licenses or into the public domain. ### Citation Information ``` @ONLINE{wikidump, author = "Wikimedia Foundation", title = "Wikimedia Downloads", url = "https://dumps.wikimedia.org" } ``` ### Contributions Thanks to [@albertvillanova](https://huggingface.co/albertvillanova) for adding this dataset.

提供机构：

wikimedia

原始信息汇总

数据集概述

语言支持

支持多种语言，包括但不限于阿拉伯语（ar）、阿萨姆语（as）、阿塞拜疆语（az）等。

许可证

数据集遵循的许可证包括：cc-by-sa-3.0 和 gfdl。

大小分类

数据集根据大小分为四个类别：n<1K, 1K<n<10K, 10K<n<100K, 100K<n<1M。

任务分类

主要任务类别包括：text-generation 和 fill-mask。
具体任务ID包括：language-modeling 和 masked-language-modeling。

数据集详细信息

配置名称: 20231201.xx（xx代表不同语言代码）
特征:
- id: 数据集中的唯一标识符，数据类型为string。
- url: 数据集中的链接地址，数据类型为string。
- title: 数据集中的标题，数据类型为string。
- text: 数据集中的文本内容，数据类型为string。
分割:
- train: 训练集，包含具体的数据量（num_bytes）和示例数量（num_examples）。
- 示例数据量（num_bytes）和下载大小（download_size）以及数据集大小（dataset_size）均详细列出。

示例数据量统计

以20231201.ar为例，训练集数据量为1027384499字节，包含38235个示例，下载大小为471633595字节，数据集总大小为1027384499字节。
其他语言的数据量统计类似，均详细列出了训练集的数据量和示例数量。

搜集汇总

数据集介绍

构建方式

Wikisource数据集源自维基媒体基金会旗下的维基文库项目，旨在收录自由内容的历史文献与原始文本。该数据集以2023年12月1日的快照为基准，通过系统化抓取各语言子站点的结构化数据构建而成。每个条目包含唯一标识符、原始URL、文档标题及正文内容，形成以语言为划分维度的独立配置（config），覆盖阿拉伯语、英语、中文等70余种语言。数据规模因语言而异，从不足千条到数十万条不等，整体采用统一的JSON格式存储，便于跨语言研究与多任务处理。

特点

该数据集的核心特点在于其多语言覆盖的广度与历史文献的深度，囊括了从古典拉丁语到现代印地语等跨越时空的文本资源。数据以语言为独立子集，每个子集仅含单次划分（train），无额外验证或测试集，适合用于无监督学习场景。文本内容忠实呈现原始文献风貌，未经过滤或改写，保留了历史语境与语言多样性。此外，数据集遵循CC-BY-SA 3.0与GFDL许可协议，确保了学术使用的合法性与可追溯性。

使用方法

在HuggingFace框架下，用户可通过指定语言标签（如'en'或'zh'）直接加载对应配置，例如使用`load_dataset('wikimedia/wikisource', '20231201.en')`获取英语子集。数据集适用于文本生成与掩码语言建模任务，尤其适合训练多语言语言模型或进行跨语言文本分析。由于仅提供训练分割，研究者需自行划分验证集以评估模型性能。数据以字符串形式存储，可直接用于深度学习框架的tokenizer处理，无需额外解析步骤。

背景与挑战

背景概述

维基文库（Wikisource）作为维基媒体基金会旗下的多语种自由文本库，自2003年启动以来，致力于收录与保存公有领域及自由授权的原始文献，涵盖历史典籍、法律条文、文学作品与学术著作等。该数据集由维基媒体社区持续维护，其核心研究问题在于如何为自然语言处理领域提供大规模、多语种、结构化的历史文本资源，以促进跨语言信息检索、文本生成与语言建模等任务的进展。自发布以来，该数据集因其丰富的语言覆盖（超过70种语言）与高质量文本内容，成为语言学、数字人文及机器翻译领域的重要基准，尤其为低资源语言的模型训练提供了稀缺的语料支撑。

当前挑战

该数据集面临的核心挑战在于语料规模的极端不均衡性：英语、俄语等语种拥有数十亿字节的文本，而孟加拉语、巽他语等仅含数千字节，导致低资源语言模型训练时数据匮乏。构建过程中，文献的数字化与校对依赖志愿者贡献，不同语种文本的OCR错误、格式不一致及版权状态模糊性增加了清洗难度。此外，部分古语或方言文本（如那不勒斯语）的现代语言模型适配性差，以及跨语种对齐任务的缺失，限制了其在多语言迁移学习中的直接应用。时间跨度大的文献还面临术语演变与历史拼写差异带来的语义理解挑战。

常用场景

经典使用场景

Wikimedia/Wikisource数据集汇聚了来自维基文库的海量多语种历史文献与公共领域文本，覆盖阿拉伯语、中文、英语、俄语等逾七十种语言。其经典使用场景在于为跨语言自然语言处理任务提供高质量的语料基础，尤其适用于语言建模与掩码语言建模。研究者可基于该数据集训练大规模语言模型，捕捉不同历史时期文本的句法结构与语义特征，从而推动多语种文本生成与理解技术的纵深发展。

实际应用

在实际应用中，Wikimedia/Wikisource数据集可支撑古籍数字化整理与智能检索系统的构建。例如，它可用于开发面向历史文献的自动摘要工具、多语种翻译系统以及文本校对平台。此外，该数据集在文化遗产保护领域价值凸显，能辅助构建数字图书馆中的知识图谱，实现跨时代、跨语言的文本关联与可视化探索，赋能教育、出版及博物馆等行业的智能化转型。

衍生相关工作

该数据集衍生了多项经典研究工作，包括基于其多语种文本训练的开源语言模型（如多语种BERT变体），以及针对历史文本的特定领域预训练模型。学者还利用其构建了跨语言文本分类与情感分析基准，推动了低资源语言评测集的标准化。此外，Wikisource数据催生了数字人文学科中文本遗传学与版本校勘的研究，为文献学与计算机科学的交叉融合提供了实证基础。

以上内容由遇见数据集搜集并总结生成