QiyaoMa/Personalized-RewardBench

---
license: apache-2.0
configs:
- config_name: Art_and_Entertainment
data_files:
- split: test
path: Art_and_Entertainment/test-*
- config_name: Lifestyle_and_Personal_Development
data_files:
- split: test
path: Lifestyle_and_Personal_Development/test-*
- config_name: Society_and_Culture
data_files:
- split: test
path: Society_and_Culture/test-*
dataset_info:
- config_name: Art_and_Entertainment
features:
- name: id
dtype: string
- name: question
dtype: string
- name: profile
list:
- name: category
dtype: string
- name: id
dtype: string
- name: text
dtype: string
- name: rubric_aspects
list:
- name: aspect
dtype: string
- name: evidence
dtype: string
- name: reason
dtype: string
- name: narrative
dtype: string
- name: category
dtype: string
- name: chosen
dtype: string
- name: rejected
dtype: string
splits:
- name: test
num_bytes: 89236554
num_examples: 767
download_size: 51586581
dataset_size: 89236554
- config_name: Lifestyle_and_Personal_Development
features:
- name: id
dtype: string
- name: question
dtype: string
- name: profile
list:
- name: category
dtype: string
- name: id
dtype: string
- name: text
dtype: string
- name: rubric_aspects
list:
- name: aspect
dtype: string
- name: evidence
dtype: string
- name: reason
dtype: string
- name: narrative
dtype: string
- name: category
dtype: string
- name: chosen
dtype: string
- name: rejected
dtype: string
splits:
- name: test
num_bytes: 88293415
num_examples: 989
download_size: 50696927
dataset_size: 88293415
- config_name: Society_and_Culture
features:
- name: id
dtype: string
- name: question
dtype: string
- name: profile
list:
- name: category
dtype: string
- name: id
dtype: string
- name: text
dtype: string
- name: rubric_aspects
list:
- name: aspect
dtype: string
- name: evidence
dtype: string
- name: reason
dtype: string
- name: narrative
dtype: string
- name: category
dtype: string
- name: chosen
dtype: string
- name: rejected
dtype: string
splits:
- name: test
num_bytes: 113883119
num_examples: 1074
download_size: 64200581
dataset_size: 113883119
---

# Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

[**📜 Paper**](https://arxiv.org/abs/2604.07343) | [**🤗 Benchmark**](https://huggingface.co/datasets/QiyaoMa/Personalized-RewardBench) | [**🖥️ Code**](https://github.com/Martin-qyma/Personalized-RewardBench)

## Abstract
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94\%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.
<img src='benchmark.png' />

---

## Dataset Structure

| Key | Type | Description |
| :--- | :--- | :--- |
| `id` | string | Unique instance identifier. |
| `question` | string | Target question prompting the response. |
| `profile` | list[dict] | User's past question history (contains `id`, `category`, `text`). Can be optionally included in the prompt so that RMs can infer the user's preferences from past history. |
| `rubric_aspects` | list[dict] | Scoring criteria extracted from the `narrative` (`aspect`, `evidence`, `reason`). Strictly **excluded from RM input** to prevent leakage of the scoring standard. |
| `narrative` | string | User's supplementary explanation of the question. Strictly **excluded from RM input** to prevent leakage of the scoring standard. |
| `category` | string | Topic domain of the question and profile. |
| `chosen` | string | Preferred, high-quality generated response. |
| `rejected` | string | Suboptimal generated response used as a negative contrast. |

## Reward Model Evaluation

During preference evaluation, `rubric_aspects` and `narrative` are strictly omitted to prevent data leakage. Only the prompt context and target responses (`chosen` / `rejected`) are passed to the reward model.

### Pointwise (Discriminative RM)
* **Input:** `question` (+ `profile`) + `chosen` / `rejected`
* **Output:** scalar reward score

### Pairwise (Generative RM)
* **Input:** `question` (+ `profile`) + `chosen` + `rejected`
* **Output:** preference label

---

## Loading the Dataset

```python
from datasets import load_dataset

# Load one configuration
ds = load_dataset("QiyaoMa/Personalized-RewardBench", name="Art_and_Entertainment")
test = ds["test"]

# Access a record
record = test[0]
print(record["question"])
print(record["chosen"])
print(record["rejected"])
```

Load all three configurations:

```python
CONFIGS = [
"Art_and_Entertainment",
"Lifestyle_and_Personal_Development",
"Society_and_Culture",
]

for config in CONFIGS:
ds = load_dataset("QiyaoMa/Personalized-RewardBench", name=config)["test"]
print(f"{config}: {len(ds)} rows")
```

---