TokenHaven/FineWeb-Edu-Norwegian

这是一个包含大量高质量挪威语文本数据及其元数据的集合。数据集由英语通用爬虫数据过滤而来，使用教育评分4或以上的FineWeb-Edu分类器进行筛选。数据来源于HuggingFaceFW/fineweb-edu数据集的v1.0.0版本，对应于通用爬虫的CC-MAIN-2024-10。数据经过去重和标签化，用于主题和格式，并保留特定格式的文档。所有文档都使用Qwen3-235B-A22B LLM模型从英语翻译成挪威语，并移除了网络抓取痕迹，使用Markdown格式重新排版以提高可读性，确保文本的高质量和清洁。如果文档没有标题，LLM还会生成标题。

This dataset contains a large collection of high-quality Norwegian text data with their metadata. The dataset was created by translating English text from the Common Crawl dataset using an LLM model. The text data is filtered for quality using classifiers and organized into various formats and topics. All documents are translated from English to Norwegian and formatted using Markdown to improve readability, ensuring high quality and cleanliness of the text. The LLM also generates a title for documents without one.