Statewise Dataset

This upload is for the Statewise Dataset, a collection of n-grams scraped from Twitter biographies collected from 2012-2023. Users have been bucketed into 53 distinct regions, covering all 50 U.S. states, D.C., Puerto Rico, and those who were identified as being within the U.S. without enough information to assign them into one of the previous categories.

Each file is an .sqlite database representing a snapshot of users observed during that year. The main table name is "bio_tokens". Within the table are the following 5 columns (all lowercase):

state: the region where the underlying user was assigned based on the contents of their location field

token: the n-gram in question

incidence: the raw count of this n-gram among users in the same state during that year

prevalence: a measurement of n-gram frequency expressed as "X per 10,000 users". Calculated by dividing incidence by num_accounts and multiplying by 10,000.

num_accounts: the total number of user accounts identified to that region within that year.

More details about the data and our analysis can be found in the full paper:

TODO: INCLUDE PAPER LINK