hacker-news-rss
收藏资源简介：
# Hacker News RSS Feed Directory **TL;DR** — We visited every unique domain ever posted to Hacker News, found which ones publish RSS/Atom feeds, and packaged the results as monthly parquet snapshots with rich metadata. **623,957** feeds discovered across **1,755,955** hosts, spanning 232 months from 2006-10 to 2026-03. *Last updated: 2026-04-05T09:21:39Z* ## Why this exists RSS is not dead — it's just hard to discover. The `<link rel="alternate">` tag that points to a site's feed is buried in HTML `<head>`, invisible to users. Feed auto-discovery requires fetching every homepage, parsing the HTML, and often probing well-known paths like `/feed`, `/rss.xml`, `/atom.xml` when the link tag is missing. Hacker News is the best proxy we have for "interesting sites on the internet." Over its 18+ year history, hundreds of thousands of unique domains have been submitted. This dataset answers a simple question: **which of those sites have RSS feeds, and what do we know about them?** ## What you get Each row = one host with a validated feed, enriched with: - **Feed metadata** — URL, type (RSS/Atom), title, description, language, generator, last published date, item count - **Site metadata** — HTML `<title>`, meta description, generator, language, favicon - **OpenGraph** — og:title, og:description, og:image, og:site_name, og:locale - **HN context** — story count, total score, first/last seen timestamps for that host in the given month Hosts without a valid feed are excluded. The feed must parse with [gofeed](https://github.com/mmcdole/gofeed) and contain at least one item. ## File structure ``` data/ ├── 2006/ │ ├── 2006-10.parquet ← first month of HN │ └── ... ├── 2024/ │ ├── 2024-01.parquet │ └── ... └── 2026/ └── ... stats.csv ← per-month scan metrics README.md ``` Each parquet file is ZSTD-compressed and sorted by `hn_story_count DESC`. ## Schema | Column | Type | Description | |--------|------|-------------| | `host` | string | Lowercase hostname (e.g. `simonwillison.net`) | | `feed_url` | string | Discovered feed URL | | `feed_type` | string | `rss`, `atom`, or `unknown` | | `feed_title` | string | Feed title | | `feed_description` | string | Feed description or subtitle | | `feed_language` | string | Feed language tag | | `feed_generator` | string | Feed generator (e.g. `Hugo 0.121.0`) | | `feed_last_pub` | timestamp | Most recent item published date | | `feed_item_count` | int | Items in feed at scan time | | `site_title` | string | HTML `<title>` | | `site_description` | string | Meta description | | `site_generator` | string | Meta generator | | `site_language` | string | `<html lang>` or meta language | | `site_favicon` | string | Favicon URL | | `og_title` | string | OpenGraph title | | `og_description` | string | OpenGraph description | | `og_image` | string | OpenGraph image URL | | `og_site_name` | string | OpenGraph site name | | `og_locale` | string | OpenGraph locale | | `hn_story_count` | int | Stories from this host in the month | | `hn_total_score` | int | Sum of HN points for the month | | `hn_first_seen` | timestamp | Earliest story time in the month | | `hn_last_seen` | timestamp | Latest story time in the month | | `scan_time` | timestamp | When this host was probed | | `month` | string | `YYYY-MM` partition key | ## Feeds discovered by year ``` 2006 █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 11 feeds (28 hosts) 2007 ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2,638 feeds (9,126 hosts) 2008 ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 7,677 feeds (26,992 hosts) 2009 ██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 12,662 feeds (44,217 hosts) 2010 ████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 19,340 feeds (66,440 hosts) 2011 ████████████████████████░░░░░░░░░░░░░░░░ 28,438 feeds (107,087 hosts) 2012 ████████████████████████████░░░░░░░░░░░░ 32,813 feeds (112,505 hosts) 2013 █████████████████████████████░░░░░░░░░░░ 34,641 feeds (108,925 hosts) 2014 ███████████████████████████░░░░░░░░░░░░░ 32,464 feeds (103,471 hosts) 2015 █████████████████████████████░░░░░░░░░░░ 34,729 feeds (106,076 hosts) 2016 ██████████████████████████████░░░░░░░░░░ 35,537 feeds (104,215 hosts) 2017 ██████████████████████████████░░░░░░░░░░ 35,644 feeds (102,162 hosts) 2018 ██████████████████████████████░░░░░░░░░░ 35,911 feeds (95,701 hosts) 2019 ████████████████████████████████░░░░░░░░ 37,892 feeds (96,110 hosts) 2020 ███████████████████████████████████████░ 45,813 feeds (117,867 hosts) 2021 ████████████████████████████████████████ 46,382 feeds (115,979 hosts) 2022 ██████████████████████████████████░░░░░░ 40,534 feeds (94,062 hosts) 2023 ████████████████████████████████████░░░░ 41,828 feeds (98,036 hosts) 2024 ████████████████████████████████████░░░░ 42,236 feeds (101,207 hosts) 2025 █████████████████████████████████████░░░ 43,855 feeds (109,331 hosts) 2026 ███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 12,912 feeds (36,418 hosts) ``` ## Quick start ### DuckDB (fastest) ```sql -- Everything, everywhere, all at once SELECT * FROM 'hf://datasets/open-index/hacker-news-rss/data/*/*.parquet'; -- Top 20 feeds for January 2024 SELECT host, feed_url, feed_title, hn_story_count FROM 'hf://datasets/open-index/hacker-news-rss/data/2024/2024-01.parquet' ORDER BY hn_story_count DESC LIMIT 20; -- Most-submitted feed hosts of all time SELECT host, feed_url, SUM(hn_story_count) AS total_stories FROM 'hf://datasets/open-index/hacker-news-rss/data/*/*.parquet' GROUP BY host, feed_url ORDER BY total_stories DESC LIMIT 50; -- What blogging platforms power the HN front page? SELECT CASE WHEN site_generator ILIKE '%wordpress%' THEN 'WordPress' WHEN site_generator ILIKE '%ghost%' THEN 'Ghost' WHEN site_generator ILIKE '%hugo%' THEN 'Hugo' WHEN site_generator ILIKE '%jekyll%' THEN 'Jekyll' WHEN site_generator ILIKE '%next%' THEN 'Next.js' WHEN site_generator ILIKE '%gatsby%' THEN 'Gatsby' WHEN site_generator ILIKE '%11ty%' OR site_generator ILIKE '%eleventy%' THEN 'Eleventy' WHEN site_generator != '' THEN site_generator ELSE 'Unknown' END AS platform, COUNT(DISTINCT host) AS sites FROM 'hf://datasets/open-index/hacker-news-rss/data/*/*.parquet' GROUP BY platform ORDER BY sites DESC LIMIT 15; -- Build your own OPML: export all feeds as XML SELECT '<outline text="' || feed_title || '" xmlUrl="' || feed_url || '" htmlUrl="https://' || host || '"/>' FROM 'hf://datasets/open-index/hacker-news-rss/data/2024/*.parquet' WHERE feed_url != '' GROUP BY host, feed_url, feed_title; ``` ### Python ```python from datasets import load_dataset ds = load_dataset("open-index/hacker-news-rss") df = ds["train"].to_pandas() # Top feeds by HN engagement top = df.groupby("host")["hn_story_count"].sum().sort_values(ascending=False) print(top.head(20)) ``` ### Python (DuckDB) ```python import duckdb df = duckdb.sql(""" SELECT host, feed_url, feed_type, SUM(hn_story_count) AS total FROM 'hf://datasets/open-index/hacker-news-rss/data/*/*.parquet' GROUP BY ALL ORDER BY total DESC LIMIT 100 """).df() print(df) ``` ## How it's built 1. **Extract** — Query the full HN archive (public ClickHouse replica) for story URLs grouped by month. Normalize hostnames, deduplicate, skip known non-feed hosts (github.com, youtube.com, twitter.com, etc.) 2. **Discover** — For each host, fetch the homepage and scan `<head>` for `<link rel="alternate" type="application/rss+xml">` or `atom+xml` tags. If nothing found, probe well-known paths: `/feed`, `/rss`, `/rss.xml`, `/atom.xml`, `/feed.xml`, `/index.xml`, and others. 3. **Validate** — Parse each candidate with gofeed. Must return at least one item. This eliminates false positives from HTML error pages served at feed paths. 4. **Enrich** — Extract HTML meta tags, OpenGraph properties, and feed-level metadata from the homepage and feed XML. 5. **Cache** — Results are cached locally (JSON). Hosts already probed — whether they have feeds or not — are skipped on re-runs. This makes incremental scanning fast. 6. **Publish** — Write ZSTD-compressed parquet via DuckDB, commit to this HuggingFace repo with stats tracking for resumable backfill. 20 concurrent workers, 10s timeout per host. Polite but thorough. ## What's not included - **Feed content** — This is a directory, not a mirror. Use `feed_url` to subscribe or fetch current items. - **Non-story URLs** — Only HN stories (Show HN, Ask HN, and regular submissions with URLs) are considered. Comments linking to sites are not. - **Sites that block bots** — Some sites return 403/503 to our user agent. They won't appear in the dataset even if they have feeds. - **IP-based hosts** — Numeric hostnames are skipped. - **The usual suspects** — GitHub, YouTube, Twitter/X, Reddit, Wikipedia, Amazon, and other aggregators/platforms are excluded since their feeds are well-known or not useful at the domain level. ## License [Open Data Commons Attribution License (ODC-BY 1.0)](https://opendatacommons.org/licenses/by/1-0/). Source data from the [HN public API](https://github.com/HackerNews/API).
