TweetLingDiv

This dataset is a resource developed to support research on digital linguistic diversity. Quantitative research in the field is limited by the lack of openly available spatiotemporal data to quantify digital language use. While there are multiple dimensions of digital linguistic diversity, such as consumption, this dataset focuses specifically on language production, here measured as the number of geotagged tweets produced. To construct the dataset, we queried all tweet-IDs in a given country and year from the UNT Geotweet Archive (Feng et al., 2023), and counted the number of language tags assigned by the Twitter language identification model, which we ordered by year and country. All in all, the dataset spans 70 languages, 236 countries and 10 years from 2012 to 2021. The data was enriched with ISO639-3 language codes and structured and merged into a single CSV: n_tweets_country_year_language.csv. Based on this, we precompute three commonly used naive measures of diversity: Richness, Exponent-Shannon, and Inverse-Simpson for each country and year, yielding the timeseries diversity_measures.csv. In addition, we compute non-naive versions of the above mentioned measures using lexical similarity based on the ASJP (Wichmann et al, 2025). Furthermore, additional data from Glottolog (Hammarström et al, 2026) and rnaturalearthdata concerning the countries and languages included in the dataset are provided in country_data.csv and language_data.csv to facilitate analysis. Initial analysis show that the number of available tweets are very low for much of the developing world, yielding uncertain diveristy estimates. Furthermore, we comfirm multiple issues with twitters language identification model, yielding very improbable language distributions in a number of countries. The data and precalcualted measures should therefore be interpreted with caution. This research was funded by WWTF (grant numberICT23-012). It is a part of the DIGILINGDIV-project.