gluclose-D/nepali-news-scraped
收藏资源简介：
--- language: - ne license: cc-by-nc-4.0 tags: - news - low-resource-language - nepali-nlp - text-classification - scraped - text-generation pretty_name: Scraped Nepali News Corpus for Low-Resource NLP size_categories: - 10K<n<100K --- # Scraped Nepali News Corpus for Low-Resource NLP ## Dataset Description This dataset is a curated collection of text scraped from prominent Nepalese news portals, including **Kantipur, Gorkhapatra, Setopati**, and others. Nepali (`ne`) is historically classified as a **low-resource language** in Natural Language Processing (NLP) due to the scarcity of high-quality, cleanly formatted, and large-scale textual corpora available for training modern machine learning models. This repository aims to bridge that gap by providing a clean, tabular dataset of contemporary Devanagari text suitable for training tokenizer vocabularies, fine-tuning language models (like BERT, DeBERTa, or LLaMA variants via QLoRA), and evaluating downstream text classification tasks. ### Dataset Details - **Language:** Nepali (नेपाली) - **Script:** Devanagari - **Data Source Portals:** Kantipur (इकान्तिपुर), Gorkhapatra (गोरखापत्र), Setopati (सेतोपाटी), and alternative mainstream portals. - **Format:** .txt --- ## Intended Uses & Applications This corpus is highly valuable for researchers and developers working on South Asian language technologies, specifically targeting: 1. **Custom Tokenization:** Training subword tokenizers (Byte-Pair Encoding, Unigram) to optimize token efficiency for the Nepali script. 2. **Masked Language Modeling (MLM):** Pre-training or domain-adapting Transformer models on clean Nepali prose. 3. **Downstream Classification:** Categorizing news into sectors such as Politics (राजनीति), Sports (खेलकुद), Economy (अर्थतन्त्र), and Entertainment (मनोरञ्जन). 4. **Text Summarization:** Using pairs of titles and body text to fine-tune sequence-to-sequence models. --- ## Data Cleaning & Preprocessing Notice * **Encoding:** Fully standardized to `UTF-8` to preserve accurate Devanagari Unicode rendering without corruption. * **Noise Reduction:** Standard web scrapers introduce HTML tags, Javascript fragments, and navigation breadcrumbs; this dataset has undergone structural filtering to ensure only raw, readable news text remains. ## Licensing & Attribution This dataset is intended primarily for **academic and research purposes** under the Creative Commons Attribution-NonCommercial 4.0 International (`cc-by-nc-4.0`) license. The intellectual property of the individual articles belongs entirely to their respective original publishers (Kantipur Media Group, Gorkhapatra Sansthan, Setopati Media, etc.). Users of this dataset must respect the copyright guidelines of the original news portals.
This dataset is a curated collection of text scraped from prominent Nepalese news portals, including Kantipur, Gorkhapatra, Setopati, and others. Nepali is historically classified as a low-resource language in Natural Language Processing due to the scarcity of high-quality, cleanly formatted, and large-scale textual corpora. This repository bridges that gap by providing a clean, tabular dataset of contemporary Devanagari text suitable for training tokenizer vocabularies, fine-tuning language models (like BERT, DeBERTa, or LLaMA variants via QLoRA), and evaluating downstream text classification tasks. The dataset is in Nepali language, uses Devanagari script, is formatted as .txt files, and has been standardized to UTF-8 encoding with noise reduction to retain only raw, readable news text.
