Curlie Dataset - Language-agnostic Website Embedding and Classification
收藏资源简介：
**************** Full Curlie dataset **************** This dataset contains the URL scrapped from curlie.org alongside with their multilingual labels. The label correspond to the sub-category where the URL was referenced in Curlie. We also provide a mapping between english labels and labels from other languages for alignment. The URLs have been filtered to only contain homepages. Each distint URL is indexed with a unique identifier (uid). curlie.csv.gz > [url, uid, label, lang] x 2,275,150 samples mapping.json.gz > [english_label, matchings] x 35,946 labels **************** Processed Curlie dataset **************** You find here the data used to train Homepage2vec. URLs have been further filtered out: websites listed under the Regional top-category where dropped, as well as non-accessible websites. This filtering yields 1,018,207 valid URL. The labels are aligned across languages and reduced to the 14 top-categories (classes). Because a URL can belong to several classes, a binary vector is used. The grouping yields 885,582 distinct URL, for each of them we provide the HTML content. We also provide a visual encoding, it was obtained by forwarding a screenshot of the homepage trough a ResNet deep-learning model pretrained on ImageNet. The training and testing sets are also given. curlie_filtered.csv.gz > [url, uid, label, lang] x 1,018,207 samples class_vector.json.gz > [uid, class_vector] x 885,582 samples class_names.txt > [class_name] x 14 classes html_content.json.gz > [uid, html] x 885,582 samples visual_encoding.json.gz > [uid, visual_encoding] x 885,582 samples train_uid.txt > [uid] x 797,023 samples test_uid.txt > [uid] x 88,559 samples **************** Pretrained Homepage2Vec**************** h2v_1000_100.zip > Model pretrained on all features h2v_1000_100_text_only.zip > Model pretrained only on textual features (no visual features from screenshots) **************** Enriched Curlie dataset **************** Thanks to Homepage2Vec, we release an enriched version of Curlie. Each URL is associated to a class probability vector and to an embedding in the latent space. outputs.json.gz > [url, uid, prediction, embedding] x 885,582 samples ******** Notes ******** JSON files have one record per line and can be read with Pandas: e.g pandas.read_json(file, orient='records', lines=True, compression='gzip')
**************** 完整Curlie数据集 **************** 本数据集包含从curlie.org爬取的URL及其多语言标签。标签对应该URL在Curlie中被归类的子类别。我们还提供了英语标签与其他语言标签的对齐映射表。已对URL进行过滤，仅保留主页链接。每个唯一URL均配有唯一标识符（unique identifier，简称uid）。 curlie.csv.gz：包含[URL、uid、标签、语言]字段，共计2,275,150条样本 mapping.json.gz：包含[英语标签、匹配映射]字段，共计35,946个标签 **************** 处理后的Curlie数据集 **************** 本部分包含用于训练Homepage2vec的数据集。进一步对URL进行了筛选：移除了属于区域顶级分类的网站，以及无法访问的网站。经此筛选后共得到1,018,207条有效URL。标签已完成跨语言对齐，并精简为14个顶级分类（类别）。由于单个URL可归属多个类别，故采用二进制向量进行标注。经归类后共得到885,582个唯一URL，我们为每个URL提供了对应的HTML内容。我们还提供了视觉编码：通过将主页截图输入在ImageNet上预训练的ResNet深度学习模型生成。同时提供了训练集与测试集。 curlie_filtered.csv.gz：包含[URL、uid、标签、语言]字段，共计1,018,207条样本 class_vector.json.gz：包含[uid、类别二进制向量]字段，共计885,582条样本 class_names.txt：包含14个类别名称，共计14行 html_content.json.gz：包含[uid、HTML内容]字段，共计885,582条样本 visual_encoding.json.gz：包含[uid、视觉编码]字段，共计885,582条样本 train_uid.txt：包含训练集的唯一标识符，共计797,023条 test_uid.txt：包含测试集的唯一标识符，共计88,559条 **************** 预训练Homepage2vec模型 **************** h2v_1000_100.zip：基于全特征预训练的模型 h2v_1000_100_text_only.zip：仅基于文本特征（无截图视觉特征）预训练的模型 **************** 增强版Curlie数据集 **************** 依托Homepage2vec，我们发布了增强版Curlie数据集。每个URL均配有类别概率向量与隐空间嵌入向量。 outputs.json.gz：包含[URL、uid、预测结果、嵌入向量]字段，共计885,582条样本 ******** 备注 ******** JSON文件每行存储一条记录，可通过Pandas读取：例如使用pandas.read_json(file, orient='records', lines=True, compression='gzip')
