NbAiLab/aurora-robots-txt
收藏Hugging Face2026-03-25 更新2026-03-29 收录
下载链接：
https://hf-mirror.com/datasets/NbAiLab/aurora-robots-txt
下载链接
链接失效反馈官方服务：
资源简介：
Fetched robots.txt for creation of the Aurora dataset.
Columns are
- `url`: The URL of the robots.txt file fetched
- `content`: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all. Pages that were tried but gave other errors contain a null value.
- `fetched`: The timestamp of the visit.
提供机构：
NbAiLab
