Curlie Dataset - Language-agnostic Website Embedding and Classification

**************** Full Curlie dataset **************** Curlie.org is presented as the largest human-edited directory of the Web. It contains 3M+ multilingual webpage classified in a hierarchical taxonomy that is language-specific, but regrouping the same 14 top-level categories. Unfortunately, the Curlie administrators do not provide a downloadable archive of this valuable content. Therefore, we decided to release our own dataset that results from a in-depth scrapping of the Curlie website. This dataset contains webpages URL alongside with the category path (label) where they are referenced in Curlie. For example, the International Ski Federation website (www.fis-ski.com) is referenced under the category path Sports/Winter/Sports/Skiing/Associations. The category path is language-specific and we provide a mapping between english and other languages for alignment. The URLs have been filtered to only contain homepages (URL with empty path). Each distinct URL is indexed with a unique identifier (uid). curlie.csv.gz > [url, uid, label, lang] x 2,275,150 samples mapping.json.gz > [english_label, matchings] x 35,946 labels **************** Processed Curlie dataset **************** We provide here the ground data used to train Homepage2Vec. URLs have been further filtered out: websites listed under the Regional top-category are dropped, as well as non-accessible websites. This filtering yields 933,416 valid entries. The labels are aligned across languages and reduced to the 14 top-categories (classes). There are 885,582 distinct URLs, for which the associated classes are represented with a binary class vector (an URL can belong to multiple classes). We provide the HTML content for each distinct URL. We also provide a visual encoding, it was obtained by forwarding a screenshot of the homepage trough a ResNet deep-learning model pretrained on ImageNet. Finally, we provide the training and testing sets for reproduction concerns. curlie_filtered.csv.gz > [url, uid, label, lang] x 933,416 samples class_vector.json.gz > [uid, class_vector] x 885,582 samples html_content.json.gz > [uid, html] x 885,582 samples visual_encoding.json.gz > [uid, visual_encoding] x 885,582 samples class_names.txt > [class_name] x 14 classes train_uid.txt > [uid] x 797,023 samples test_uid.txt > [uid] x 88,559 samples **************** Enriched Curlie dataset **************** Thanks to Homepage2Vec, we release an enriched version of Curlie. For each distinct URL, we provide the class probability vector (14 classes) and the latent space embedding (100 dimensions). outputs.json.gz > [uid, url, score, embedding] x 885,582 samples **************** Pretrained Homepage2Vec**************** h2v_1000_100.zip > Model pretrained on all features h2v_1000_100_text_only.zip > Model pretrained only on textual features (no visual features from screenshots) **************** Notes **************** CSV file can be read with python: import pandas as pd df = pd.read_csv(“curlie.csv.gz“, index_col=0) JSON files have one record per line and can be read with python: import json import gzip with gzip.open("html_content.json.gz", "rt", encoding="utf-8") as file: for line in file: data = json.loads(line) …**************** 完整Curlie数据集 **************** Curlie.org 是目前规模最大的人工编辑式网页目录。该平台收录超300万条多语言网页，采用语言专属的层级分类体系，但所有语言版本均共享同一套共14个顶级分类。遗憾的是，Curlie官方并未提供该珍贵内容的可下载归档包。因此，我们通过对Curlie官网进行深度爬取，构建并发布了自研的Curlie数据集。本数据集收录网页URL及其在Curlie平台中对应的分类路径（标签）。例如，国际滑雪联合会官网（www.fis-ski.com）的分类路径为Sports/Winter/Sports/Skiing/Associations（体育/冬季运动/滑雪/协会）。分类路径具有语言特异性，我们同时提供了英文与其他语言标签的对齐映射表。所有URL均经过过滤，仅保留首页（路径为空的URL）。每个唯一URL均配有唯一标识符（uid）。 curlie.csv.gz：包含[网页URL、唯一标识符、分类标签、语言]字段，共计2,275,150条样本 mapping.json.gz：包含[英文标签、对应匹配项]字段，共计35,946个标签 **************** 预处理版Curlie数据集 **************** 本数据集为训练Homepage2Vec模型所用的基准数据。本次进一步对URL进行了过滤：移除了顶级分类为「地区」的网站，同时剔除了无法访问的站点。经此过滤后，共得到933,416条有效条目。所有标签已完成跨语言对齐，并仅保留14个顶级分类（类别）。本数据集包含885,582个唯一URL，其关联类别以二元类别向量表示（一个URL可归属多个类别）。我们为每个唯一URL提供了对应的HTML内容。此外我们还提供了视觉编码结果：该编码通过将首页截图输入在ImageNet（图像分类基准数据集）上预训练的ResNet（残差神经网络）深度学习模型生成。为便于实验复现，我们同时提供了训练集与测试集。 curlie_filtered.csv.gz：包含[网页URL、唯一标识符、分类标签、语言]字段，共计933,416条样本 class_vector.json.gz：包含[唯一标识符、类别向量]字段，共计885,582条样本 html_content.json.gz：包含[唯一标识符、HTML内容]字段，共计885,582条样本 visual_encoding.json.gz：包含[唯一标识符、视觉编码结果]字段，共计885,582条样本 class_names.txt：包含[类别名称]字段，共计14个类别 train_uid.txt：包含训练集唯一标识符，共计797,023条样本 test_uid.txt：包含测试集唯一标识符，共计88,559条样本 **************** 增强版Curlie数据集 **************** 依托Homepage2Vec模型，我们发布了增强版Curlie数据集。针对每个唯一URL，我们提供了14个类别的类别概率向量，以及100维的隐空间嵌入向量。 outputs.json.gz：包含[唯一标识符、网页URL、类别概率、隐空间嵌入]字段，共计885,582条样本 **************** 预训练Homepage2Vec模型 **************** h2v_1000_100.zip：基于全特征训练的预训练模型 h2v_1000_100_text_only.zip：仅基于文本特征训练的预训练模型（未使用截图生成的视觉特征） **************** 使用说明 **************** CSV文件可通过Python读取，示例代码如下： import pandas as pd df = pd.read_csv("curlie.csv.gz", index_col=0) JSON文件每行存储一条记录，可通过Python读取，示例代码如下： import json import gzip with gzip.open("html_content.json.gz", "rt", encoding="utf-8") as file: for line in file: data = json.loads(line) …