Gloogle web scraping dataset

This is the full SQLite database used in Gloogle, an experimental search engine focused on indexing a healthier slice of the web. At the time of writing, it includes around 1 million rows. All data was crawled in July 2025, but newer versions might go after that. The database's main table, pages, includes the following columns: id url title keywords snippet derank created_at and updated_at (timestamps in ISO format) There's also an index for SQLite full-text search, pages_fts. All data was collected using a Bun-powered web crawler, seeded with a handpicked list of starter URLs, consisting of mostly small personal sites, niche wikis, and independent blogs and following all links that match the criteria below. Many of these came from webrings, and almost half of the dataset is wikipedia for whatever reason. Indexing criteria: Blocked domains/TLDs: t.me, .i2p, .onion, marxists.org (causes infinite loops due to broken 404s), web.archive.org, x.com, twitter.com, pro.imdb.com, sherdog.com (way too many pages that aren't really relevant) Blocked file extensions: .jpg, .jpeg, .png, .gif, .pdf, .epub, .doc, .docx, .zip, .rar, .mp3, .mp4, .exe, .dmg Blocked URL paths: /download/, /pdf/, /login, /static/, /video/, /audio/, /rss, /feed, /vote/up, /vote/down, /trending/ And pages that explicitly state they are not in English or that ask crawlers not to index them. Note that the crawler doesn't respect robots.txt, but it does respect noindex, nofollow metatags or links, and the X-Robots-Tag header. Pages are “deranked” if they embed trackers, ad scripts, or third-party iframes that are in the known derank list. The derank count reflects how many such elements were found. The crawler should avoid nsfw, but I haven't fully audited the dataset. The actual search engine powered by this dataset is at gloogle.tiagorangel.com