big-banyan-tree/BBT_CommonCrawl_2021

BigBanyanTree项目旨在帮助大学建立数据工程集群，并推动使用Apache Spark等工具进行数据处理和分析的兴趣。该数据集是这一倡议的直接结果，由Gautam和Suchit在Harsh Singhal的指导下处理。数据集内容来源于2021-49 CommonCrawl dump的885个随机采样的WARC文件，每个arrow文件包含从这些WARC文件中提取的字段。此外，数据集使用MaxMind的GeoLite2-City_20240903数据库来丰富WARC数据的地理位置信息。

The BBT-CC21 dataset originates from the BigBanyanTree project, which aims to help colleges set up data engineering clusters and drive interest in data processing and analysis using tools like Apache Spark. The dataset, processed by Gautam and Suchit under the guidance of Harsh Singhal, contains fields extracted from Common Crawl WARC files and is enriched with geolocation information using the MaxMind database. The dataset includes data processed from 885 randomly sampled WARC files from the 2021-49 CommonCrawl dump. It is important to note that the data may contain inaccuracies or outdated information, and no validation or filtering has been performed on the extracted URLs or IP addresses, which may include objectionable content. Use the dataset with caution, especially for tasks involving personal or sensitive information.