Repeated Survey Responses of Large Language Models: A Dataset for Measuring Intra-Model Variability and Ideological Bias (Russian-Language Context, Kazakhstan)

Description

This dataset contains repeated responses generated by large language models (LLMs) to a standardized questionnaire designed to measure ideological orientations and value-based attitudes. The data were collected within an experimental framework aimed at assessing both inter-model differences and intra-model variability under controlled conditions.

The dataset includes responses from three models: ChatGPT (n = 101), Alice (n = 99), and Copilot (n = 32). All interactions were conducted in Russian using identical prompts, ensuring comparability across models and iterations. Each model completed the same set of scale-based questions multiple times, allowing for the analysis of response dispersion, stability, and distributional characteristics.

The questionnaire consists of 12 indicators (Q1–Q10c) capturing attitudes toward key socio-political dimensions, including democracy, leadership, redistribution, state responsibility, law compliance, traditions, national identity, and institutional trust. The dataset is structured in a tabular format, where each row represents a single model run and each column corresponds to a survey item.

The primary purpose of the dataset is to enable the study of:

intra-model variability (consistency of responses within the same model),

inter-model differences (systematic variation across models),

distributional properties of responses beyond mean values,

potential ideological bias and its stability across repeated elicitation.

The dataset supports quantitative analysis using nonparametric statistical methods and resampling techniques, including the estimation of variability indices such as the Intra-Model Variability (IMV) metric and bootstrap-based confidence intervals.

This resource is particularly relevant for research in computational social science, AI evaluation, and algorithmic bias, providing an empirical basis for assessing reproducibility and stability in generative model outputs. The approach of repeated survey elicitation offers a replicable framework for examining how LLMs respond to socially sensitive content over multiple interactions, addressing limitations of single-response evaluation designs.