hacker-news-text-search

# Hacker News text + substring patterns Sampled comments and stories from the full year 2025 of the public [Hacker News](https://news.ycombinator.com) archive, paired with small curated dictionaries of substring patterns and precomputed match labels. The intended use is testing text-search and substring-matching code on real, messy English text: multi-byte characters, HTML entities, embedded URLs, mixed casing, CVE identifiers, version strings, and the long tail of forum slang. ## Layout at a glance - `text_pool/comments.{parquet,csv}` - 3,398,696 comments, the canonical text store. Columns: `text_id, ts, parent, by, text`. - `text_pool/stories.{parquet,csv}` - 310,384 stories. Columns: `story_id, ts, title, url, by, score`. - `pattern_pool/{topics,categories,domains,bans,allow}.{parquet,csv}` - curated pattern dictionaries reused across workloads. - One directory per workload (see table below). Each carries `patterns.{parquet,csv}`, `labels.{parquet,csv}`, `meta.json`, and (for the HQ multi-relation queries) `query.sql`. The text side is not re-emitted: every workload joins back to `text_pool/` so the baseline is unified across the dataset - resolve `labels.text_id` against `text_pool/comments.parquet` (or `text_pool/stories.parquet` for workloads marked `stories` in the table below). Both Parquet and CSV are shipped side by side: the HF dataset viewer reads the Parquet files (faster, typed); the CSVs make raw `wget` / `curl` downloads trivial for tooling that does not speak Parquet. ## Source and license Pulled from the [`open-index/hacker-news`](https://huggingface.co/datasets/open-index/hacker-news) mirror, partitions `data/2025/*.parquet` (full year 2025). After filtering `deleted=0 AND dead=0` and stripping HTML, the slice carries 3,709,080 rows: 3,398,696 comments and 310,384 stories. Redistributed under CC BY-SA 4.0, matching the upstream mirror. Attribution: Hacker News (Y Combinator) and the [`open-index`](https://huggingface.co/open-index) maintainers of the upstream mirror. ## Workloads | config | kind | text side | description | size | |--------|------|-----------|-------------|------| | `hq1_topic_mentions` | hq | comments | HQ1: per-category comment mention count over a topic watchlist; paper analog of the Audio Promo query. | 50,000 texts, 10 matches | | `hq1_topic_mentions_edge` | hq_edge | comments | HQ1 LIKE edge: 50k comments x 100 topic patterns | 50,000 texts, M=100, 14,774 matches | | `hq2_domain_topic` | hq | comments | HQ2: comments on stories whose URL matches a domain pattern, joined to topic patterns. Two LIKE edges; the engine has to pick a join order. | 50,000 texts, 5 matches | | `hq2_topic_edge` | hq_edge | comments | HQ2 topic LIKE edge after URL-domain pushdown: surviving comments x 100 topic patterns | 31 texts, M=100, 9 matches | | `hq2_url_edge` | hq_edge | stories | HQ2 URL LIKE edge: 10k stories x 20 domain patterns | 9,444 texts, M=20, 1,088 matches | | `hq3_hot_stories` | hq | comments | HQ3: top 50 stories by comment-topic density. LIKE edge feeds a COUNT(*) that drives the ORDER BY. | 50,000 texts, 19 matches | | `hq3_hot_stories_edge` | hq_edge | comments | HQ3 LIKE edge: 50k comments x 30 topic patterns (cats 0,4,2) | 50,000 texts, M=30, 2,263 matches | | `hq4_author_distinct` | hq | comments | HQ4: authors whose comments touch >= 3 distinct topics. COUNT(DISTINCT topic_id) requires AC to emit pattern identity. | 50,000 texts, 1,138 matches | | `hq4_author_distinct_edge` | hq_edge | comments | HQ4 LIKE edge: 50k comments x 100 topic patterns (identical shape to hq1_edge; emitted separately for table layout) | 50,000 texts, M=100, 14,774 matches | | `hq5_timeline` | hq | comments | HQ5: daily mention counts per topic over 2025-04. Exercises filter pushdown of ts before the LIKE-join. | 50,000 texts, 54 matches | | `hq5_timeline_edge` | hq_edge | comments | HQ5 LIKE edge: 50k comments x 20 topic patterns (cats 2,4). ts filter targets 2025-04 slice. | 50,000 texts, M=20, 921 matches | | `hq6_allow_edge` | hq_edge | comments | HQ6 allow LIKE edge: 50k comments x 30 allow patterns | 50,000 texts, M=30, 1,837 matches | | `hq6_ban_allow` | hq | comments | HQ6: count comments matching a ban list but not the allow list. Models a trust-and-safety filter; AC can do ban+allow in one pass. | 50,000 texts, 1 matches | | `hq6_bans_edge` | hq_edge | comments | HQ6 bans LIKE edge: 50k comments x 31 ban patterns | 50,000 texts, M=31, 194 matches | | `q1_trending_M10` | q | comments | Q1: %TOKEN% substring on HN comments; ASCII single-literal patterns scaling in M. | 20,000 texts, M=10, 697 matches | | `q1_trending_M100` | q | comments | Q1: %TOKEN% substring on HN comments; ASCII single-literal patterns scaling in M. | 20,000 texts, M=100, 146,159 matches | | `q1_trending_M1k` | q | comments | Q1: %TOKEN% substring on HN comments; ASCII single-literal patterns scaling in M. | 20,000 texts, M=1000, 554,943 matches | | `q2_phrase_M100` | q | comments | Q2: %a%b% multi-segment phrase patterns on HN comments. | 20,000 texts, M=100, 3,078 matches | | `q2_phrase_M30` | q | comments | Q2: %a%b% multi-segment phrase patterns on HN comments. | 20,000 texts, M=30, 2,870 matches | | `q3_positional_adversarial` | q | comments | Q3: `_` positional constraints; family=adversarial | 4,968 texts, M=5, 10,281 matches | | `q3_positional_cves` | q | comments | Q3: `_` positional constraints; family=cves | 287 texts, M=4, 752 matches | | `q3_positional_dates` | q | comments | Q3: `_` positional constraints; family=dates | 3,235 texts, M=4, 3,711 matches | | `q3_positional_versions` | q | comments | Q3: `_` positional constraints; family=versions | 6,316 texts, M=7, 894 matches | | `q4_unicode` | q | comments | Q4: multi-byte UTF-8 literals + `_` straddling code-point boundaries. | 5,000 texts, M=20, 22 matches | | `q5_escape` | q | comments | Q5: ESCAPE-clause patterns over HTML-entity-bearing comment bodies. | 10,000 texts, M=10, 7,308 matches | | `q6_ilike_ilike` | q | comments | Q6: camelCase tech names with ILIKE. | 10,000 texts, M=20, 643 matches | | `q6_ilike_like` | q | comments | Q6: camelCase tech names with LIKE. | 10,000 texts, M=20, 317 matches | | `q7_url` | q | stories | Q7: anchored & interior URL patterns over story URLs. | 20,000 texts, M=10, 2,617 matches | | `q8_scan_literal_long` | q | comments | Q8: single-pattern scan (literal_long). | 50,000 texts, M=1, 11 matches | | `q8_scan_literal_short` | q | comments | Q8: single-pattern scan (literal_short). | 50,000 texts, M=1, 555 matches | | `q8_scan_two_seg` | q | comments | Q8: single-pattern scan (two_seg). | 50,000 texts, M=1, 546 matches | | `q8_scan_underscore` | q | comments | Q8: single-pattern scan (underscore). | 50,000 texts, M=1, 54 matches | Two families are present: - **Q-family** (`q1` ... `q8`): flat `texts x patterns` pairs. Single relation on each side. Good for isolating substring matchers from any surrounding SQL. - **HQ-family** (`hq1` ... `hq6`): multi-relation SQL queries embedding one or two `LIKE` joins inside realistic shapes (GROUP BY, TOP-K, HAVING COUNT DISTINCT, EXISTS / NOT EXISTS). `hq*_edge` configs are the same data flattened to the Q-family layout, exposing only the `LIKE` edge. ## Quick look in Python ```python from datasets import load_dataset # Browse the canonical text store. comments = load_dataset( "open-index/hacker-news-text-search", "comments", split="train" ) print(comments[0]) # Pick one workload's patterns and labels, join against the pool. patterns = load_dataset( "open-index/hacker-news-text-search", "q1_trending_M100", split="patterns", ) labels = load_dataset( "open-index/hacker-news-text-search", "q1_trending_M100", split="labels", ) print(len(patterns), "patterns;", len(labels), "matches") ``` ## Sampling Per-workload subsets are picked by BLAKE2b over `(workload_id, row_id)` so the same workload always picks the same rows across re-runs. No global RNG, no `random.sample`. ## Redactions A small list of `text_id`s is excluded from `text_pool` because the comment body matches a ClamAV exploit-PoC signature (the user quoted a real PoC verbatim). The list is in `redactions.json` at the dataset root and replicated under `meta.json` of every workload. As of this revision: 1 row dropped. ## Reproducing locally Code that downloads the upstream parquet, cleans it in DuckDB, and emits this dataset lives at [`tamnd/hackernews-processing`](https://github.com/tamnd/hackernews-processing). Re-running the pipeline against the same upstream parquet at the same pinned month reproduces every byte of this dataset.