heihei/llm-psychometric-response-style

---
license: cc-by-4.0
task_categories:
- text-classification
- tabular-classification
language:
- en
tags:
- llm-psychology
- personality-assessment
- response-style
- psychometric
- likert-scale
size_categories:
- 1K<n<10K
---

# LLM Psychometric Response Style Dataset

This dataset accompanies the paper **"The Missing Trade-Off: How LLMs Lose Human-Like Personality Structure"** (EMNLP 2026 Findings).

## Overview

We administered 61 Likert-scale psychometric items (9 dimensions) to 33 LLMs from 15 model families, collecting response style data across multiple experimental conditions.

## Files

### `main_data.json` (490 records)

Combined data from Studies 1-5:

| Study | Description | Records |
|-------|-------------|---------|
| Study 1 | Cross-model comparison (15 models × 12 seeds) | 180 |
| Study 2 | Within-family evolution (18 model versions) | 204 |
| Study 3 | Aligned vs. base comparison (3 families) | 49 |
| Study 4 | Thinking mode ablation (8 models) | 36 |
| Study 5 (direct missing) | Direct prompt condition for GLM-5 and Ring | 24 |

### `study5_prompt_sensitivity.json` (540 records)

Prompt sensitivity experiment: 15 models × 3 prompt variants × 12 seeds each.

| Variant | Description |
|---------|-------------|
| `neutral` | Instruction removed, "rate your agreement" |
| `persona` | Human role-play: "you are completing a personality survey" |
| `direct` | No instruction, only the item and scale |

## Record Format

Each record contains:

```json
{
"model_id": "Qwen/Qwen3.5-397B-A17B",
"vendor": "qwen",
"arch": "moe",
"study": 1,
"seed": 1,
"thinking_mode": "chat",
"timestamp": "2026-03-20T12:00:00",
"items": {
"bfi_extraversion": [3, 2, 3, 3, 3, 1, 3, 3],
"bfi_agreeableness": [4, 4, 1, ...],
"...": "..."
},
"bfi.extraversion": 2.625,
"bfi.agreeableness": 3.111,
"...": "...",
"response_stats": {
"valid_count": 61,
"mean": 2.91,
"sd": 0.35
}
}
```

## Dimensions

| Dimension | Items | Source |
|-----------|-------|--------|
| Extraversion | 8 | BFI-2 |
| Agreeableness | 9 | BFI-2 |
| Conscientiousness | 9 | BFI-2 |
| Neuroticism | 8 | BFI-2 |
| Openness | 10 | BFI-2 |
| Honesty-Humility | 5 | HEXACO-PI-R |
| Collectivism | 4 | Schwartz Values Survey |
| Intuition | 4 | Cognitive Style |
| Uncertainty Avoidance | 4 | Cultural Dimensions |

## Citation

```bibtex
@inproceedings{llm-psychology-2026,
title={The Missing Trade-Off: How LLMs Lose Human-Like Personality Structure},
author={...},
booktitle={Findings of EMNLP 2026},
year={2026}
}
```

## License

CC-BY-4.0