ChatGPT Early Adoption in Higher Education: Variation in Student Usage, Instructional Support and Educational Equity

Data for this study were collected
at the University of California – Irvine (UCI) as part of the UCI-MUST
(Measuring Undergraduate Success Trajectories) Project, a larger longitudinal
measurement project aimed at improving understanding of undergraduate experience,
trajectories and outcomes, while supporting campus efforts to improve
institutional performance and enhance educational equity (Arum et. al.
2021). The project is focused on
student educational experience at a selective large, research-oriented public
university on the quarter system with half of its students first-generation and
85 percent Hispanic, Asian, African-American, Pacific Islander or Native
American. Since Fall 2019, the project
has tracked annually new cohorts of freshmen and juniors with longitudinal
surveys administered at the end of every academic quarter. Data from the Winter 2023 end of term
assessment, administered in the first week of April, was pooled for four
longitudinal study cohorts for this study (i.e., Fall 2019-2022 cohorts). There
was an overall response rate of 42.5 percent for the Winter 2023 end of term
assessment. This allowed us to consider student responses from freshmen through
senior years enrolled in courses throughout the university.

Students completed questionnaire
items about their knowledge and use of ChatGPT in and out of the classroom
during the winter 2023 academic term. In total 1,129 students completed the
questionnaire, which asked questions about: knowledge of ChatGPT (“Do you know
what ChatGPT is?”); general use (“Have you used ChatGPT before?”); and
instructor attitude (“What was the attitude of the instructor for [a specific
course students enrolled in] regarding the use of ChatGPT?”). Of those 1,129
students, 191 had missing data for at least one variable of interest and were
subsequently dropped from analysis, resulting in a final sample of 938
students.<br>

In addition, for this study we merged our
survey data with administrative data from campus that encompasses details on
student background, including gender, race, first-generation college-going, and
international student status. Campus
administrative data also provides course-level characteristics, including
whether a particular class is a lower- or upper-division course as well as the
academic unit on campus offering the course.
In addition, we used administrative data on all students enrolled at the
university to generate classroom composition measures for every individual
course taken by students in our sample – specifically the proportion of
underrepresented minority students in the class, the proportion of
international students in the class and the proportion of female students in
the class.

For our student-level analysis [R1],
we used binary logistic regressions to examine the association between
individual characteristics and (1) individual awareness and (2) individual
academic use of ChatGPT utilizing the student-level data of 938 students.
Individual characteristics include gender, underrepresented minority student
status, international student status, first generation college-going student
status, student standing (i.e. lower or upper classmen), cumulative grade point
average and field of study. Field of
study was based on student major assigned to the broad categories of physical
sciences (i.e. physical sciences, engineering, and information and computer
science), health sciences (i.e. pharmacy, biological sciences, public health,
and nursing), humanities, social sciences (i.e. business, education, and social
sciences), the arts, or undeclared. We defined awareness of ChatGPT as an
affirmative response to the question “Do you know what ChatGPT is?” Regarding
ChatGPT use, we focused on academic use which was defined as an affirmative
response of either “Yes, for academic use” or “Yes, for academic and personal
use” to the question “Have you used ChatGPT before?”

For our course-level analysis [R2],
we constructed a measure – course-level instructor
encouragement for ChatGPT use – based on student responses to the end of the
term survey conducted at the completion of the Winter 2023 term. In the survey,
students were asked to indicate the extent to which their instructors encouraged
them to use ChatGPT in each of their enrolled courses. The response options
were: (1), "very much discouraged"; (2), "somewhat
discouraged"; (3), "neither discouraged nor encouraged"; (4),
"somewhat encouraged"; and (5), "very much encouraged." Our
study generated data on 57 percent of all standard course sections offered in
Winter 2023, excluding those without a classroom component, such as independent
studies or study abroad. For course
sections mentioned multiple times by students in the sample, we aggregated
student responses at the course level (courses in our analysis on average had
10.4 students reporting on instructor encouragement or discouragement of
ChatGPT use). As a result, our analysis encompassed 1,047 distinct course sections reported by
915 students, with an average encouragement score of 2.7 (out of 5).
For the course-level analysis, we conducted simple binary regressions to
examine the relationships between average student-reported instructor
encouragement in specific courses and the proportion of underrepresented
minority students, as well as the proportion of international students in the
class. Scatterplots were created to visualize these relationships, with fitted
regression lines included to highlight general trends. Both the scatterplot
markers and the regression models were weighted by classroom size to ensure
that larger classes are more influential than smaller classes in
our course-level statistical analyses. <br><b><br>Citation:</b> Arum, Richard, Eccles, Jacquelynne, Heckhausen,
Jutta, Von Keyserlingk, Luise, Li, XunFei, Yu, Renzhe, Orona, G., Mathew, D.,
Chang, D. UCI Measuring Undergraduate Success Trajectories (UCI-MUST) Project
Data, [United States], 2019-2025. <br>