匈牙利语通用文本语料库

本数据集面向多语言大模型研发中对中东欧语言的支持需求，解决匈牙利语这类中等资源语言高质量语料匮乏的问题。提供3.1亿条匈牙利语文本，涵盖通用新闻、学术论文及行政文档等类型。 数据规模足以支撑训练数亿参数的匈牙利语专用NLP模型，或为多语言大模型提供中东欧语言的增量预训练语料。清洗流程针对匈牙利语的高黏着性、元音和谐及18个格变位进行了语素兼容处理。This dataset targets the demand for Central and Eastern European (CEE) language support in the development of multilingual large language models (LLMs), addressing the scarcity of high-quality corpora for mid-resource languages such as Hungarian. It provides 310 million Hungarian text instances spanning multiple genres including general news, academic papers, and administrative documents. The scale of this dataset is adequate to support the training of Hungarian-specific NLP models with hundreds of millions of parameters, or to supply incremental pre-training corpora for CEE languages to enhance multilingual LLMs. The data cleaning pipeline has implemented morpheme-compatible processing tailored to the highly agglutinative nature, vowel harmony, and 18 case inflections of the Hungarian language.

搜集汇总

背景与挑战

背景概述

该数据集旨在支持多语言大模型研发，解决匈牙利语作为中等资源语言高质量语料不足的问题，提供3.1亿条涵盖通用新闻、学术论文和行政文档的文本。其402GB的规模可训练匈牙利语专用NLP模型或用于多语言大模型的增量预训练，且清洗流程针对匈牙利语的高黏着性、元音和谐及18个格变位进行了优化处理。