Cross-Lingual Dataset of Crisis-Related Social Media

The cross-lingual natural disaster dataset includes public tweets collected using Twitter’s public API, filtering by location-related keywords and date, without using any additional filtering (e.g., we did not restrict the query to specific languages). We considered two disaster events and two long-term natural disasters across Europe (floods and wildfires) that received substantial news coverage internationally.

Three of the top languages were common to all the studied events: English (ISO 639-1 code: en), Spanish (es), and French (fr). Additionally, we found hundreds of messages for each event in other five languages, including Arabic (ar), German (de), Japanese (ja), Indonesian (id), Italian (it) and Portuguese (pt). 

After collecting the data, we labelled tweets that contained potentially informative factual information. We name this group of tweets “informative messages.” Next, we used crowdsourcing to further categorize the messages into various informational categories. We asked three different workers to label each informative messages across languages. The target categories were based on an ontology from TREC-IS 2018, where we grouped some low level ontology categories into higher-level ones.