Curlie Dataset - Language-agnostic Website Embedding and Classification
收藏资源简介：
**************** Full Curlie dataset **************** This dataset contains the URL scrapped from curlie.org alongside with their multilingual labels. The label correspond to the sub-category where the URL was referenced in Curlie. We also provide a mapping between english labels and labels from other languages for alignment. The URLs have been filtered to only contain homepages. Each distint URL is indexed with a unique identifier (uid). curlie.csv.gz > [url, uid, label, lang] x 2,275,150 samples mapping.json.gz > [english_label, matchings] x 35,946 labels **************** Processed Curlie dataset **************** You find here the data used to train Homepage2vec. URLs have been further filtered out: websites listed under the Regional top-category where dropped, as well as non-accessible websites. This filtering yields 1,018,207 valid URL. The labels are aligned across languages and reduced to the 14 top-categories (classes). Because a URL can belong to several classes, a binary vector is used. The grouping yields 885,582 distinct URL, for each of them we provide the HTML content. We also provide a visual encoding, it was obtained by forwarding a screenshot of the homepage trough a ResNet deep-learning model pretrained on ImageNet. The training and testing sets are also given. curlie_filtered.csv.gz > [url, uid, label, lang] x 1,018,207 samples class_vector.json.gz > [uid, class_vector] x 885,582 samples class_names.txt > [class_name] x 14 classes html_content.json.gz > [uid, html] x 885,582 samples visual_encoding.json.gz > [uid, visual_encoding] x 885,582 samples train_uid.txt > [uid] x 797,023 samples test_uid.txt > [uid] x 88,559 samples **************** Pretrained Homepage2Vec**************** h2v_1000_100.zip > Model pretrained on all features h2v_1000_100_text_only.zip > Model pretrained only on textual features (no visual features from screenshots) **************** Enriched Curlie dataset **************** Thanks to Homepage2Vec, we release an enriched version of Curlie. Each URL is associated to a class probability vector and to an embedding in the latent space. outputs.json.gz > [url, uid, prediction, embedding] x 885,582 samples ******** Notes ******** JSON files have one record per line and can be read with Pandas: e.g pandas.read_json(file, orient='records', lines=True, compression='gzip')
**************** 完整Curlie数据集 **************** 本数据集包含从curlie.org爬取的URL及其多语言标签。标签对应该URL在Curlie中被归类的子分类。我们还提供了英语标签与其他语言标签的对齐映射表。所有URL均经过筛选，仅保留首页。每个独立URL均配有唯一标识符（unique identifier，uid）。 curlie.csv.gz：包含[url, uid, 标签, 语言]字段，共2,275,150条样本 mapping.json.gz：包含[英语标签, 匹配项]字段，共35,946个标签 **************** 预处理Curlie数据集 **************** 本部分提供用于训练Homepage2vec的数据集。URL再次经过筛选：移除了属于区域顶级分类的网站，以及无法访问的网站。经此筛选后共得到1,018,207个有效URL。标签已实现跨语言对齐，并归类至14个顶级分类（类别）。由于单个URL可隶属于多个类别，故采用二进制向量进行标注。经分组后共得到885,582个独立URL，我们为每个URL提供其HTML内容。我们还提供了视觉编码：该编码通过将首页截图输入在ImageNet上预训练的ResNet深度学习模型生成。同时提供训练集与测试集。 curlie_filtered.csv.gz：包含[url, uid, 标签, 语言]字段，共1,018,207条样本 class_vector.json.gz：包含[uid, 类别向量]字段，共885,582条样本 class_names.txt：包含14个类别名称，每行一个类别 html_content.json.gz：包含[uid, HTML内容]字段，共885,582条样本 visual_encoding.json.gz：包含[uid, 视觉编码]字段，共885,582条样本 train_uid.txt：包含797,023个训练集uid，每行一个 test_uid.txt：包含88,559个测试集uid，每行一个 **************** 预训练Homepage2Vec模型 **************** h2v_1000_100.zip：基于全特征预训练的模型 h2v_1000_100_text_only.zip：仅基于文本特征（不包含截图的视觉特征）预训练的模型 **************** 增强版Curlie数据集 **************** 借助Homepage2Vec，我们发布了增强版Curlie数据集。每个URL均配有类别概率向量与隐空间嵌入向量。 outputs.json.gz：包含[url, uid, 预测结果, 嵌入向量]字段，共885,582条样本 ******** 备注 ******** JSON格式文件每行包含一条记录，可通过Pandas读取，示例代码：pandas.read_json(file, orient="records", lines=True, compression="gzip")
