Background data for: Ordinal response scales: Psychometric grounding for design and analysis

This dataset contains background data and supplementary material for a methodological study on the use of ordinal response scales in linguistic research. For the literature survey reported in that study, which examines how rating scales are used in current linguistic research (4,441 papers from 16 linguistic journals, published between 2012 and 2022), it includes a tabular file listing the 406 research articles that report ordinal rating scale data. This file records annotated attributes of the studies and rating scales. Further the dataset includes summary data gathered in a review of the psychometric literature on the interpretation of quantificational expressions that are often used to build graded scales. Empirical findings are collected for five rating scale dimensions: agreement (1 study), intensity (3 studies), frequency (17 studies), probability (11 studies), and quality (3 studies). Finally, the post includes new data from 20 informants on the interpretation of the quantifiers \"few\", \"some\", \"many\", and \"most\".
Abstract: Related publication Ordinal scales are commonly used in applied linguistics. To summarize the distribution of responses provided by informants, these are usually converted into numbers and then averaged or analyzed with ordinary regression models. This approach has been criticized in the literature; one caveat (among others) is the assumption that distances between categories are known. The present paper illustrates how empirical insights into the perception of response labels may inform the design and analysis stage of a study. We start with a review of how ordinal scales are used in linguistic research. Our survey offers insights into typical scale layouts and analysis strategies, and it allows us to identify three commonly used rating dimensions (agreement, intensity, and frequency). We take stock of the experimental literature on the perception of relevant scale point labels and then demonstrate how psychometric insights may direct scale design and data analysis. This includes a careful consideration of measurement-theoretic and statistical issues surrounding the numeric-conversion approach to ordinal data. We focus on the consequences of these drawbacks for the interpretation of empirical findings, which will enable researchers to make informed decisions and avoid drawing false conclusions from their data. We present a case study on yous(e) in British and Scottish English, which shows that reliance on psychometric scale values can alter statistical conclusions, while also giving due consideration to the key limitations of the numeric-conversion approach to ordinal data analysis.