TopicBased

In this work, we present a dataset, which is collected from English-language Wikipedia and Twitter users. In contrast to the corpus of existing studies, we focused on the corpus of Wikipedia, which contains many articles and covers a multitude of topics. We performed preprocessing, including lemmatizing, converting slang into common English, and removing stop words. Then the articles in the English-language version of Wikipedia were converted to plain text. The Wiki markup was processed and a text version of each item was returned. By using Tweepy [1] to access the Twitter API, we collected the data from Twitter users. We crawled the list of each user account’s followers. Next, we obtained the most recent tweets of each user in the study.

本研究构建了一款数据集，其采集自英文维基百科（Wikipedia）与推特（Twitter）用户平台。与现有研究采用的语料库不同，本研究聚焦于维基百科语料库——该语料库包含海量文章，涵盖众多主题领域。我们开展了预处理操作，包括词形还原（lemmatizing）、将俚语转换为标准英语以及移除停用词。随后将英文维基百科的文章转换为纯文本格式，对维基百科标记语言（Wiki markup）进行处理后，输出每篇文章的纯文本版本。我们借助Tweepy库[1]访问推特API（Twitter API）采集推特用户相关数据，爬取了各用户账号的粉丝列表，后续进一步获取了本研究中每位用户的最新推文内容。