Racó Forums Corpus

The Racó Forums Corpus is a 19-million-sentence corpus of Catalan user-generated text built from the forums of Racó Català. Since the existing available corpora in Catalan lacked conversational data, we searched for a major source of such data for Catalan, and we found Racó Català, a popular multitopic online forum. We obtained a database dump and we transformed all the threads so that we obtained documents that traversed all the existing paths from the root (initial comment) to the leaves (last comment with no reply). In other words, if T is a tree such that T = {A,B,C,D} and the first comment is A that is replied by B and C independently, and, then, C is replied by D, we obtain two different documents A,B and A,C,D in the fairseq language modeling format. The corpora has been anonimized by changing user names to common Catalan names and removing emails and mentions to the forum itself. Find the README拉科论坛语料库（Racó Forums Corpus）是一个包含1900万句加泰罗尼亚语用户生成文本的语料库，数据采集自Racó Català论坛。鉴于当前已有的加泰罗尼亚语语料库均缺失对话类数据，我们针对性地搜寻加泰罗尼亚语对话数据的主流来源，最终发现了Racó Català——一个广受欢迎的多主题在线论坛。我们获取了该论坛的数据库备份，并对所有讨论串（thread）进行格式转换，最终得到覆盖从根节点（首条评论）到叶节点（无后续回复的末条评论）所有现有路径的文档。换言之，若存在树结构T={A,B,C,D}，首条评论为A，分别收到B与C的独立回复，且C又得到D的回复，则我们将得到两种符合fairseq语言建模格式的不同文档：A,B与A,C,D。该语料库已完成匿名化处理：将用户名替换为常见加泰罗尼亚人名，并移除了邮箱地址以及对该论坛的相关提及内容。详见README文档。