NbAiLab/aurora-robots-txt
收藏官方服务：
资源简介：
Fetched robots.txt for creation of the Aurora dataset. Columns are - `url`: The URL of the robots.txt file fetched - `content`: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all. Pages that were tried but gave other errors contain a null value. - `fetched`: The timestamp of the visit.
提供机构：
NbAiLab
