CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

Time-critical analysis of social media streams is important for humanitarian organizations for planing rapid response during disasters. The crisis informatics research community has developed several techniques and systems for process- ing and classifying big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature (e.g., for training models), it is not pos- sible to compare the results and measure the progress made towards building better models for crisis informatics tasks. In this work, we attempt to bridge this gap by combining various existing crisis-related datasets. We consolidate eight human-annotated datasets and provide 166.1k and 141.5k tweets for informativeness and humanitarian classification tasks, respectively. We believe that the consolidated dataset will help train more sophisticated models. Moreover, we pro- vide benchmarks for both binary and multiclass classifica- tion tasks using several deep learning architecrures including, CNN, fastText, and transformers. We make the dataset and scripts available at: https://crisisnlp.qcri.org/ crisis_datasets_benchmarks.html