big-banyan-tree/BBT_CommonCrawl_2018

BigBanyanTree项目旨在帮助大学建立数据工程集群，并推动使用Apache Spark等工具进行数据处理和分析的兴趣。该数据集是该项目的直接成果，包含从2018-51 CommonCrawl dump的900个随机抽样的WARC文件中提取的字段。数据集还使用了MaxMind的GeoLite2-City_20240903数据库来丰富WARC数据的地理位置信息。需要注意的是，数据可能包含不准确或过时的信息，且未经过滤，可能包含不当内容。

The BigBanyanTree initiative aims to empower colleges to set up their data engineering clusters and drive interest towards data processing and analysis using tools such as Apache Spark. This dataset is a direct result of this initiative, containing fields extracted from 900 randomly sampled WARC files from the 2018-51 CommonCrawl dump. The dataset is enriched with geolocation information using the MaxMind GeoLite2-City_20240903 database. It is important to note that the data may contain inaccuracies or outdated information and has not been filtered, potentially including objectionable content.