Dataset: The Odyssey of robots.txt Governance: Measuring Compliance Implications of Web Crawling Bots in Large Language Model Services

Name: Dataset: The Odyssey of robots.txt Governance: Measuring Compliance Implications of Web Crawling Bots in Large Language Model Services
Creator: Zenodo
Published: 2025-09-17T02:35:47+08:00

Zenodo2025-09-17 更新2026-05-26 收录

数据链接：

https://zenodo.org/doi/10.5281/zenodo.17138492 数据链接链接失效反馈

官方服务：

资源简介：

This artifact contains the dataset for the paper: "The Odyssey of robots.txt Governance: Measuring Compliance Implications of Web Crawling Bots in Large Language Model Services"

提供机构：

Zenodo

创建时间：

2025-09-17