nixiesearch/hackernews-stories

该数据集名为HackerNews stories dataset，基于nixiesearch/hackernews-comments数据集，主要包含从HackerNews中提取的故事（type=story）的目标URL内容。每个故事都通过trafilatura库解析，并以markdown格式存储文章的文本及页面特定的元数据。数据集的时间覆盖范围从2006年10月到2024年9月，共抓取了2150271个页面，解压后的大小约为20GB。数据集以JSONL格式文件存储，并使用ZSTD压缩。

This dataset, named HackerNews stories dataset, is based on the nixiesearch/hackernews-comments dataset and primarily contains the target URL content of stories (type=story) extracted from HackerNews. Each story is parsed using the trafilatura library, and the article text along with page-specific metadata is stored in markdown format. The dataset covers the period from October 2006 to September 2024, with a total of 2,150,271 pages scraped, and the unpacked size is approximately 20GB. The dataset is stored in JSONL format files and compressed using ZSTD.