open-index/hacker-news-text-search

该数据集包含从2025年全年Hacker News公开存档中采样的评论和故事，并配对了精心策划的子字符串模式词典和预计算的匹配标签。其主要用途是在真实、杂乱的英文文本上测试文本搜索和子字符串匹配代码，涵盖多字节字符、HTML实体、嵌入URL、混合大小写、CVE标识符、版本字符串以及论坛俚语的长尾分布。数据集分为两个主要部分：文本池（包括评论和故事）和模式池（包括主题、类别、域名、禁止和允许模式）。此外，还提供了多个工作负载配置，分为Q系列（扁平化的文本x模式对，用于隔离子字符串匹配器）和HQ系列（嵌入一个或两个LIKE连接的多关系SQL查询，模拟真实场景如GROUP BY、TOP-K等）。数据集以Parquet和CSV格式提供，便于不同工具使用。数据来源于open-index/hacker-news镜像，遵循CC BY-SA 4.0许可证。This dataset consists of sampled comments and stories from the full year 2025 of the public Hacker News archive, paired with curated dictionaries of substring patterns and precomputed match labels. It is intended for testing text-search and substring-matching code on real, messy English text, including multi-byte characters, HTML entities, embedded URLs, mixed casing, CVE identifiers, version strings, and the long tail of forum slang. The dataset is organized into two main components: a text pool (containing comments and stories) and a pattern pool (including topics, categories, domains, bans, and allow patterns). It also features multiple workload configurations, categorized into Q-family (flat texts x patterns pairs for isolating substring matchers) and HQ-family (multi-relation SQL queries embedding one or two LIKE joins within realistic shapes such as GROUP BY, TOP-K, etc.). The dataset is provided in both Parquet and CSV formats for compatibility with various tools. The data is sourced from the open-index/hacker-news mirror and is distributed under the CC BY-SA 4.0 license.