pysentimiento/spanish-tweets

一个大型的（主要是）西班牙语推文数据集，用于预训练语言模型（或其他表示）。数据集包含推文ID、用户ID和推文文本三个字段。数据集创建过程涉及从Archive.org下载Spritzer集合，并筛选出语言元数据为西班牙语的推文。数据集包含622M条推文，来自约432K用户。虽然主要是西班牙语，但也包含少量葡萄牙语、英语等其他语言的推文（约7/8%的推文不是西班牙语）。

原始信息汇总

数据集概述

数据集名称

• 名称: spanish-tweets

数据集描述

• 描述: 一个用于预训练语言模型的大型西班牙语推文数据集。
• 语言: 主要为西班牙语，包含少量葡萄牙语、英语和其他语言。

数据集结构

• 特征:
  • text: 字符串类型，推文内容。
  • tweet_id: 字符串类型，推文ID。
  • user_id: 字符串类型，用户ID。
• 分割:
  • train: 597433111条样本，总大小82649695458字节。
  • test: 6224733条样本，总大小892219251字节。
• 下载大小: 51737237106字节。
• 数据集大小: 83541914709字节。

数据集创建

• 数据收集: 从Archive.org下载的Spritzer集合中筛选出西班牙语推文，并下载相关用户的推文线。
• 数据量: 包含约622M条推文，来自约432K用户。

引用信息

@inproceedings{perez-etal-2022-robertuito, title = "{R}o{BERT}uito: a pre-trained language model for social media text in {S}panish", author = "P{e}rez, Juan Manuel and Furman, Dami{a}n Ariel and Alonso Alemany, Laura and Luque, Franco M.", booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference", month = jun, year = "2022", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://aclanthology.org/2022.lrec-1.785", pages = "7235--7243", abstract = "Since BERT appeared, Transformer language models and transfer learning have become state-of-the-art for natural language processing tasks. Recently, some works geared towards pre-training specially-crafted models for particular domains, such as scientific papers, medical documents, user-generated texts, among others. These domain-specific models have been shown to improve performance significantly in most tasks; however, for languages other than English, such models are not widely available. In this work, we present RoBERTuito, a pre-trained language model for user-generated text in Spanish, trained on over 500 million tweets. Experiments on a benchmark of tasks involving user-generated text showed that RoBERTuito outperformed other pre-trained language models in Spanish. In addition to this, our model has some cross-lingual abilities, achieving top results for English-Spanish tasks of the Linguistic Code-Switching Evaluation benchmark (LinCE) and also competitive performance against monolingual models in English Twitter tasks. To facilitate further research, we make RoBERTuito publicly available at the HuggingFace model hub together with the dataset used to pre-train it.", }