Preprocessed Apex Domains from the Domains Project for Tokenizer Training

This dataset contains 270,762,723 unique, validated, registerable (apex) domain names derived from the Domains Project (https://domainsproject.org/, BSD-3-Clause). It was used to train the SentencePiece BPE and Unigram tokenizers reported in "Tokenization of Internet Domains: Bridging NLP and Network Naming". Contents preprocessed_domains.txt — one apex domain per line, lexicographically sorted, UTF-8 encoded. Preprocessing pipeline Starting from 1,764,770,720 hostnames in the raw Domains Project release, we apply the following transformations (full pipeline reproduced in the paper, Table 1): Hostname → apex domain collapse using the Public Suffix List (1,427,894,341 hostnames collapsed; 336,858,285 unique apex domains retained). Removal of suffix-only entries (646), and malformed labels (17,448). Removal of domains under PSL private suffixes; typically machine-generated subdomains or blog pages under shared platforms (e.g. *.compute.amazonaws.com, blogspot.com, myshopify.com, herokuapp.com, github.io, vercel.app); 66,085,596 entries removed. IDNA 2008 (RFC 5891) validation with UTS #46 compatibility processing in non-transitional mode and Unicode NFC normalization (9,966 entries rejected: 8,877 emoji domains, 808 IDNA encoding failures, 275 underscored labels, 6 with invisible characters). Composition 1,429 unique TLDs covered (out of 1,590 IANA-registered). 68.3 % under generic TLDs, 30.9 % under country-code TLDs, 0.2 % sponsored, ~0.6 % other. Top-5 TLDs: .com (51.4 %), .net (4.5 %), .de (4.2 %), .org (3.7 %), .uk (2.0 %). 99.4 % pure ASCII; 1.6 M valid Internationalized Domain Names (IDNs). Intended use Training of subword tokenizers, language models, or any lexical/statistical model over the global domain name space. The dataset is deliberately disjoint from the downstream classification labels in the companion Common Crawl and Curlie releases (only 463 K domains co-occur in all three). Citation If you use this dataset, please cite the companion paper and the upstream Domains Project.