nhagar/redpajama-data-1t_urls

该数据集提供了从RedPajama-Data-1T训练记录中提取的URL和顶级域名。它旨在方便研究人员和实践者探索大型语言模型训练数据集的内容，而无需管理庞大的原始文本数据。数据集由Nick Hagar和Jack Bandy策划，并采用与源数据集相同的许可。数据集包含两列：url（每个记录的原始URL）和domain（每个URL的顶级域名）。

This dataset provides URLs and top-level domains extracted from the training records of the RedPajama-Data-1T. It is intended to facilitate the exploration of large LLM training datasets without the need to manage massive amounts of raw text data. The dataset was curated by Nick Hagar and Jack Bandy, and is licensed under the same terms as the source dataset. It includes two columns: url for the raw URL of each record, and domain for the top-level domain extracted using tldextract.