nhagar/fineweb-edu_urls

该数据集提供了从训练记录中下载的URL和顶级域名。它是由Nick Hagar和Jack Bandy策划的，旨在让研究人员和实践者能够分析大型语言模型训练数据集的内容，而无需处理庞大的文本数据。数据集包含两个列：URL（每条记录关联的原始URL）和domain（每个URL的顶级域名，使用tldextract提取）。

This dataset provides the URLs and top-level domains extracted from the training records. Curated by Nick Hagar and Jack Bandy, it aims to enable researchers and practitioners to analyze the content of large language model training datasets without dealing with massive text data. The dataset includes two columns: url (the raw URL associated with each record) and domain (the top-level domain for each URL, extracted with tldextract).