big-banyan-tree/BBT_CommonCrawl_2020

BigBanyanTree是一个旨在帮助大学建立数据工程集群并促进对数据处理和分析工具兴趣的倡议。这里提供的数据是该倡议的直接结果。数据由Gautam和Suchit在Harsh Singhal的指导下处理。每个`arrow`文件包含从Common Crawl WARC文件中提取的字段表。提供的数据集是从2020-50 CommonCrawl转储中随机抽取的895个WARC文件处理而来。用于增强WARC数据的地理位置信息的MaxMind数据库是GeoLite2-City_20240903（2024年9月3日发布）。警告部分提醒用户数据可能包含不准确或过时的信息，且未经过滤，可能包含不适当的内容。免责声明部分说明了数据的使用目的，即用于分析域名分布和IP元数据。

BigBanyanTree is an initiative to empower colleges to set up their data engineering clusters, and drive interest towards data processing and analysis using tools such as Apache Spark. The data provided here is the direct result of this initiative. The data was processed by Gautam and Suchit, under the guidance of Harsh Singhal. Each `arrow` file contains a table with fields extracted from Common Crawl WARC files. The datasets provided are derived from processing randomly sampled 895 WARC files from the 2020-50 CommonCrawl dump. The MaxMind database used to enrich WARC data with geolocation information is GeoLite2-City_20240903 (released on 3rd Sept. 2024). The WARNING section alerts users that the data may contain inaccuracies or outdated information, and has not been filtered, potentially including inappropriate content. The Disclaimer section states that the data is intended for analyzing domain distributions and IP metadata analysis.