Scripts and data for analysing websites' robots.txt

Contains Jupyter notebook scripts, historical raw robots.txt data, and statistics on this data.本数据集包含Jupyter笔记本（Jupyter notebook）脚本、历史原始robots.txt数据，以及针对上述数据的统计信息。