ThatDeveloperGuy13/llms-txt-directory-snapshot

这是一个截至2026年5月底，实现了llms.txt规范的735+个网站的快照数据集。llms.txt是一个提议的标准，允许网站在稳定URL（如/llms.txt和/llms-full.txt）上声明大型语言模型（LLM）可读的摘要，以便AI爬虫（如GPTBot、ClaudeBot、PerplexityBot）无需解析完整HTML即可获取网站的精选概述。数据集包含735个唯一网站，并附有已验证的llms.txt文件，以及约30个llms-full.txt变体，由ThatDevPro从社区维护的GitHub仓库编译并丰富。该数据集可用于训练/评估AI爬虫以发现llms.txt部署、跨行业基准测试llms.txt采用率、作为MCP-llms-txt等LLM上下文工具的种子语料库，以及研究llms.txt早期采用者的地理和行业分布。A snapshot of 735+ websites implementing the llms.txt specification as of late May 2026. llms.txt is a proposed standard for websites to declare LLM-readable summaries at a stable URL (e.g., /llms.txt and /llms-full.txt), enabling AI crawlers (such as GPTBot, ClaudeBot, PerplexityBot) to retrieve a curated overview of the site without parsing the full HTML. The dataset includes 735 unique websites with verified llms.txt files, plus an additional ~30 with llms-full.txt variants, compiled and enriched by ThatDevPro from a community-maintained GitHub repository. It can be used for training/evaluating AI crawlers to discover llms.txt deployments, benchmarking llms.txt adoption rates across industries, seeding corpus for MCP-llms-txt and similar LLM context tools, and researching the geographic and industry distribution of llms.txt early adopters.