Arpuuu/nepalitext-language-model-dataset

--- annotations_creators: - no-annotation language_creators: - found - other language: - ne license: - cc0-1.0 multilinguality: - monolingual source_datasets: - extended|oscar - extended|cc100 task_categories: - text-generation task_ids: - language-modeling pretty_name: nepalitext-language-model-dataset --- # Dataset Card for "nepalitext-language-model-dataset" ### Dataset Summary "NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: [OSCAR](https://huggingface.co/datasets/oscar) , [cc100](https://huggingface.co/datasets/cc100) and a set of scraped Nepali articles on Wikipedia. ### Supported Tasks and Leaderboards This dataset is intended to pre-train language models and word representations on Nepali Language. ### Languages The data is focused on Nepali language, but may have instances of other languages as well. ## Dataset Structure ### Data Instances An example: ``` {'text': 'घरेलु मैदानमा भएको च्याम्पियन्स लिगको दोस्रो लेगमा एथ्लेटिको मड्रिडले आर्सनललाई एक शून्यले हराउँदै समग्रमा दुई एकको अग्रताका साथ फाइनलमा प्रवेश गरेको हो ।\n'} ``` ### Data Fields The data fields are: - `text`: a `string` feature. ### Data Splits train|test| ----:|---:| 13141222|268189| ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations The dataset does not contain any additional annotations. #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models. ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information [More Information Needed] ### Citation Information If you use this dataset in your research, please cite: ```bibtex @inproceedings{maskey-etal-2022-nepali, title = "{N}epali Encoder Transformers: An Analysis of Auto Encoding Transformer Language Models for {N}epali Text Classification", author = "Maskey, Utsav and Bhatta, Manish and Bhatt, Shiva and Dhungel, Sanket and Bal, Bal Krishna", editor = "Melero, Maite and Sakti, Sakriani and Soria, Claudia", booktitle = "Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages", month = jun, year = "2022", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://aclanthology.org/2022.sigul-1.14/", pages = "106--111", abstract = "Language model pre-training has significantly impacted NLP and resulted in performance gains on many NLP-related tasks, but comparative study of different approaches on many low-resource languages seems to be missing. This paper attempts to investigate appropriate methods for pretraining a Transformer-based model for the Nepali language. We focus on the language-specific aspects that need to be considered for modeling. Although some language models have been trained for Nepali, the study is far from sufficient. We train three distinct Transformer-based masked language models for Nepali text sequences: distilbert-base (Sanh et al., 2019) for its efficiency and minuteness, deberta-base (P. He et al., 2020) for its capability of modeling the dependency of nearby token pairs and XLM-ROBERTa (Conneau et al., 2020) for its capabilities to handle multilingual downstream tasks. We evaluate and compare these models with other Transformer-based models on a downstream classification task with an aim to suggest an effective strategy for training low-resource language models and their fine-tuning." } ``` ### Contributions Thanks to [@Sakonii](https://github.com/Sakonii) for adding this dataset.

搜集汇总

背景与挑战

背景概述

NepaliText语言建模数据集是一个包含超过1300万尼泊尔语文本序列的集合，由OSCAR、cc100和维基百科尼泊尔语文章组合而成，旨在用于尼泊尔语语言模型的预训练。数据集为单语尼泊尔语，无额外标注，采用CC0-1.0许可协议。