wiki-dyk

# Wikipedia "Did You Know" Archive Structured data curated from the "Did you know" (DYK) snippets Wikipedia communities feature on the Main Page — the short, sourced "... that *X*?" facts. In DYK parlance each such fact is a **hook**; this dataset holds one record per hook occurrence, enriched with the articles the hook links to. | Config | Language | Rows | DYK section | | --- | --- | --- | --- | | `en` | English | 117,342 | *Did you know ...* | | `fr` | French | 12,011 | *Le saviez-vous ?* | | `de` | German | 14,354 | *Schon gewusst?* | The schema is identical across languages (language-specific absences are `null`), so the configs can be mixed or compared directly. ## Usage ```python from datasets import load_dataset en = load_dataset("wikimedia/wiki-dyk", "en", split="train") print(en[0]["content_body"]) ``` ## Data fields One record per hook occurrence: | Field | Type | Description | | --- | --- | --- | | `language` | string | Wiki language code (`en`, `fr`, `de`). | | `archive_page` | string | Source archive page the hook was read from. | | `day` | date | Date the hook appeared on the Main Page. `null` where the archive does not record it (French before 2017). | | `set_time` | timestamp (UTC) | Set-removal timestamp where the archive records one (English from 2008); `null` elsewhere. | | `hook_wikitext` | string | Raw wikitext of the hook, for re-rendering or re-analysis. | | `content_body` | string | Reader HTML rendered locally from the wikitext (bold, italic, links). | | `linked_pages` | list<struct> | Articles the hook links to, enriched (see below). | Each entry in `linked_pages`: | Field | Type | Description | | --- | --- | --- | | `page_id` | int64 | Page id of the canonical (redirect-resolved) target; `null` if the article was later deleted. | | `rev_id` | int64 | Revision current on the DYK date — historically accurate and reproducible. | | `page_title` | string | Canonical target title (redirects resolved), not the link text. | | `short_description` | string | Short description (local on enwiki, Wikidata-backed elsewhere); `null` if none. | | `page_image` | string | The article's free page image filename; `null` if none. | | `wikidata_qid` | string | The article's Wikidata item (e.g. `Q42`). | | `article_topics` | list<struct{qid:int64, title:string}> | Topic categories from the topictrends service; `qid` is the topic's Wikidata item (stable cross-language key), `title` its localized name. `null` when unavailable. | | `prominent` | bool | The featured article(s) of the hook — the bold link(s) on wikis with a bolding convention. | ### Example record ```json { "language": "en", "archive_page": "Wikipedia:Did you know archive/2026/July", "day": "2026-07-08", "set_time": "2026-07-08T00:00:00Z", "hook_wikitext": "* ... that '''[[Iron Bridge, Lucknow|one of the earliest iron bridges in India]]''' ''(pictured)'' was ...?", "content_body": "... that <b><a href=\"./Iron_Bridge,_Lucknow\">one of the earliest iron bridges in India</a></b> <i>(pictured)</i> was ...?", "linked_pages": [ { "page_id": 12345, "rev_id": 67890, "page_title": "Iron Bridge, Lucknow", "short_description": "Bridge in Uttar Pradesh, India", "page_image": "Iron_bridge_Lucknow.jpg", "wikidata_qid": "Q130206787", "article_topics": [{ "qid": 9079095, "title": "Bridges_in_India" }], "prominent": true } ] } ``` ## How it was built DYK archive pages are fetched as raw wikitext and parsed per-language; `content_body` is rendered locally from the parse tree (no HTML API). Linked pages are enriched via each wiki's action API: canonical title and page id (redirects resolved), the revision current on the DYK date, short description, free page image, Wikidata item, plus topic categories from the topictrends service. A hook appearing across multiple days or sets yields one record per occurrence. Source code: https://github.com/santhoshtr/dyk-archive/ ## License and attribution This dataset is derived from Wikipedia article and archive text, which is released under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) (also available under the GFDL). The dataset is distributed under the same terms. Attribution is to the Wikipedia contributors of each source wiki ([en](https://en.wikipedia.org), [fr](https://fr.wikipedia.org), [de](https://de.wikipedia.org)). Per CC BY-SA, reuse must credit Wikipedia and share adaptations under a compatible license. `page_title`, `rev_id`, and `archive_page` let any fact be traced back to its exact source revision. ## Limitations - Coverage follows each wiki's archive completeness; months with no archive page are simply absent (normal gaps, not errors). - `content_body` is a faithful local rendering, but unusual templates in old eras may render as raw wikitext in rare cases. - `day` and `set_time` are `null` where the archive format never recorded them.