big-banyan-tree/BBT_CommonCrawl_2019

BigBanyanTree项目旨在帮助大学建立数据工程集群，推动使用Apache Spark等工具进行数据处理和分析的兴趣。该数据集是由Gautam和Suchit在Harsh Singhal的指导下处理的，来源于2019-51 CommonCrawl dump的895个随机抽样的WARC文件，并使用了MaxMind的GeoLite2-City_20240903数据库来丰富WARC数据的地理位置信息。数据可能包含不准确或过时的信息，且未经过滤，可能包含不适当的内容，使用时需谨慎。

The BigBanyanTree initiative aims to empower colleges to set up their data engineering clusters and drive interest towards data processing and analysis using tools such as Apache Spark. The dataset was processed by Gautam and Suchit under the guidance of Harsh Singhal, derived from 895 randomly sampled WARC files from the 2019-51 CommonCrawl dump, and enriched with geolocation information using the MaxMind GeoLite2-City_20240903 database. The data may contain inaccuracies or outdated information and has not been filtered, potentially including inappropriate content, so caution is advised when using it.