Worth the Weight: An Examination of Unstructured and Structured Data in Graduate Admissions

In graduate admissions, as in many multiattribute decisions, evaluators must judge candidates from a flood of information, including recommendation letters, personal statements, grades, and standardized test scores. Some of this information is structured, while some is unstructured. Yet most studies of multiattribute decisions focus on decisions from structured information. This study evaluated how structured and unstructured information is used within graduate admissions decisions. We examined a uniquely comprehensive dataset of N = 2,231 graduate applications to the University of Kansas, containing full application packages, demographics, and final admissions decisions for each applicant. To make sense of our documents, we applied structural topic modeling (STM), a model that allows topic content and prevalence to covary based on other metadata (e.g., department of study). STM allowed us to examine what information the letters and statements contain, and the relationships between variables like gender and race and textual information. We found that most topics in the unstructured data related to specific fields of study. The STMs did not uncover strong differences among applicants regarding race and gender, though recommendation letters and personal statements for international applicants did show some different topic profiles than domestic applicants. We also found that admissions decision-makers behaved as if they prioritized structured numeric metrics, using unstructured information to check for disqualifications, if at all. However, we found that topics were less reliable than admissions documents, meaning that additional ways of using them cannot be completely ruled out. The implications of our findings on graduate admissions decisions are discussed.

在研究生招生过程中，如同众多多属性决策情形，评审人员需从浩如烟海的信息中甄别候选人，包括推荐信、个人陈述、成绩单以及标准化考试成绩。其中，部分信息为结构化数据，而另一部分则为非结构化数据。然而，多数关于多属性决策的研究集中于结构化信息的决策分析。本研究旨在探讨结构化与非结构化信息在研究生招生决策中的运用情况。我们分析了堪萨斯大学2,231份研究生申请的全套申请材料、人口统计学数据以及每位申请者的最终录取决定。为解读我们的文档，我们采用了结构化主题建模（STM）方法，该方法允许主题内容及其频度根据其他元数据（例如，研究领域）相互关联。STM技术使我们能够考察信件和陈述中所包含的信息，以及性别、种族等变量与文本信息之间的关系。我们发现，非结构化数据中的大多数主题与特定研究领域相关。结构化主题建模并未揭示申请者在种族和性别方面的显著差异，尽管国际申请者的推荐信和个人陈述显示出与国内申请者不同的主题特征。此外，我们发现招生决策者似乎更重视结构化的量化指标，并利用非结构化信息以确认资格的缺失，即便是在有限的情况下。然而，我们发现主题的可靠性不如录取文件，这意味着无法完全排除以其他方式使用它们的可能性。我们对研究生招生决策的影响进行了深入讨论。