Common Crawl Domain-Level Multi-Label Language Identification Benchmark

A multi-label language-identification benchmark for Internet apex domain names, derived from the Common Crawl columnar URL index CC-MAIN-2026-04. Each example pairs an apex domain string with a 30-dimensional binary indicator vector over languages. To our knowledge this is the first openly released domain-level (rather than page- or sentence-level) language-ID dataset at this scale. Contents (JSON Lines) File Examples train.jsonl 24,053,427 val.jsonl 6,872,407 test.jsonl 3,436,205 metadata.json full schema, statistics, preprocessing parameters Each line: {"text": "nanohap.eu", "class_vector": [0,0,0,0,0,1,0, ... ,0]} where class_vector[i] == 1 if the domain belongs to class metadata["classes"][i]. Label space (30 languages, ISO 639-3) ara, ces, dan, deu, ell, eng, fin, fra, heb, hun, ind, ita, jpn, kor, lat, nld, nor, pol, por, ron, rus, slk, spa, srp, swe, tha, tur, ukr, vie, zho — spanning 8 script families (Latin, Cyrillic, CJK, Arabic, Greek, Hangul, Hebrew, Thai). Preprocessing pipeline Parse 2,924,038,459 page-level CLD2 detections covering 64,647,914 hostnames from the CC-MAIN-2026-04 columnar URL index. Collapse hostnames to apex domains (PSL); filter IP addresses (88,214), suffix-only entries (1,158), malformed entries (35), PSL-private suffixes (530,983), IDNA-invalid entries (38). For each apex domain, compute the proportion of URLs CLD2 assigned to each language. Apply threshold τ = 0.10: each language above τ becomes a positive label. Of the 161 languages discovered, retain the 30 with ≥ 100,000 domain-language occurrences (full list of 131 dropped languages is recorded in metadata.json). Drop 156,909 domains with no language above τ. Random 70 / 20 / 10 train/val/test split, seed 42. Statistics 35,049,969 unique apex domains processed → 34,362,039 final examples. Average label cardinality: 1.3605 (66.0 % single-label, 29.8 % double-label). Class imbalance ≈ 237× (English 25.1 M vs. Hebrew 106 K). Intended use Training and evaluation of (a) multi-label language identification on short, whitespace-free strings, and (b) tokenizer-quality probes (see companion paper). Limitations Labels are proxy labels: CLD2 classifies the HTML content of crawled pages, not the domain string itself. CLD2 misclassifications propagate; we mitigate this with the τ = 0.10 threshold and a 100 K-occurrence frequency floor. The 131 languages dropped below this threshold are listed in metadata.json for full transparency. Attribution Derived from Common Crawl Foundation data (https://commoncrawl.org/). Please cite Common Crawl in addition to this dataset.