big-banyan-tree/BBT_CommonCrawl_2023

BigBanyanTree项目旨在帮助大学建立数据工程集群，并推动使用Apache Spark等工具进行数据处理和分析的兴趣。本数据集是该项目的直接成果，由Gautam和Suchit在Harsh Singhal的指导下处理。数据集内容来源于2023-50 CommonCrawl dump的900个随机采样的WARC文件，并使用了MaxMind的GeoLite2-City_20240903数据库来丰富地理位置信息。每个`arrow`文件包含从Common Crawl WARC文件中提取的字段表。数据集中的URL和IP地址来源于公开的Common Crawl数据转储，可能包含不准确或过时信息，且未经过滤，可能包含不良内容，使用时应谨慎。

The BBT-CC23 dataset is produced by the BigBanyanTree initiative, aimed at helping colleges set up data engineering clusters and drive interest in data processing and analysis using tools like Apache Spark. The dataset contains fields extracted from Common Crawl WARC files, enriched with geolocation information using the MaxMind database. The dataset includes two configurations: script_extraction and ipmaxmind, corresponding to different data files. The dataset is sourced from 900 randomly sampled WARC files from the 2023-50 CommonCrawl dump. It is important to note that the data may contain inaccuracies or outdated information, and no validation or filtering has been performed on the extracted URLs and IP addresses, which may include objectionable content. Use this data with caution, especially for tasks involving personal or sensitive information.