cfilt/IITB-IndicMonoDoc
收藏资源简介：
这是一个包含22种印度计划语言和英语的单语种文档级语料库，总共有39518.51百万个标记。语料库包括的语言有：阿萨姆语、孟加拉语、古吉拉特语、印地语、卡纳达语、克什米尔语、贡根语、马拉雅拉姆语、曼尼普尔语、马拉地语、尼泊尔语、奥里亚语、旁遮普语、梵语、信德语、泰米尔语、泰卢固语、乌尔都语、博多语、桑塔利语、迈蒂利语和 Dogri 语。
This is a monolingual document-level corpus for 22 scheduled languages of India plus English, totaling 39518.51 million tokens. The corpus includes the following languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, Urdu, Bodo, Santhali, Maithili, and Dogri.
数据集概述
数据集名称
IITB Document level Monolingual Corpora for Indian languages
包含语言
- 主要语言：Assamese, Bengali, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, Urdu, Bodo, Santhali, Maithili, Dogri
- 其他语言：English
数据集规模
- 总规模：39,518.51 million tokens
- 各语言规模（部分示例）：
- Bengali: 5,258.47 million tokens
- English: 11,986.53 million tokens
- Gujarati: 887.18 million tokens
- Hindi: 11,268.33 million tokens
- Kannada: 567.16 million tokens
- Malayalam: 845.32 million tokens
- Marathi: 1,066.76 million tokens
- Nepali: 1,542.39 million tokens
- Punjabi: 449.61 million tokens
- Tamil: 2,171.92 million tokens
- Telugu: 767.18 million tokens
- Urdu: 2,391.79 million tokens
许可证
CC-BY-4.0
任务类别
- text-generation
标签
- language-modeling
- llm
- clm
引用信息
@misc{doshi2024worry, title={Do Not Worry if You Do Not Have Data: Building Pretrained Language Models Using Translationese}, author={Meet Doshi and Raj Dabre and Pushpak Bhattacharyya}, year={2024}, eprint={2403.13638}, archivePrefix={arXiv}, primaryClass={cs.CL} }
