big-banyan-tree/BBT_CommonCrawl_2022
收藏Hugging Face2024-10-11 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/big-banyan-tree/BBT_CommonCrawl_2022
下载链接
链接失效反馈官方服务：
资源简介：
BigBanyanTree项目旨在帮助大学建立数据工程集群，并推动使用Apache Spark等工具进行数据处理和分析的兴趣。数据集的内容是从Common Crawl WARC文件中提取的字段，这些数据来源于2022-49 CommonCrawl dump的900个随机抽样的WARC文件，并使用MaxMind数据库增加了地理位置信息。数据集中的URL和IP地址来源于公开的Common Crawl数据转储，可能存在不准确或过时的信息，且未经过滤，可能包含不适当的内容。
The BigBanyanTree initiative aims to empower colleges to set up their data engineering clusters and drive interest towards data processing and analysis using tools such as Apache Spark. The dataset contains fields extracted from Common Crawl WARC files, sourced from 900 randomly sampled WARC files from the 2022-49 CommonCrawl dump, enriched with geolocation information using the MaxMind database. The URLs and IP addresses in the dataset are sourced from publicly available Common Crawl data dumps, which may contain inaccuracies or outdated information and have not been filtered, potentially including inappropriate content.
提供机构：
big-banyan-tree
