RecoReact

# Overview
RecoReact is a novel dataset of multi-turn interactions between real users and a (random) AI assistant, providing a unique set of feedback signals, including multi-turn natural language requests, structured item selections, item ratings, and user profiles.
The dataset spans three different domains: news articles, travel destinations, and meal planning.

## Dataset Details

This dataset contains 1785 interactions from 595 users (with a median completion time of about 22 min). RecoReact also includes the user profile data of each user (fully anonymized), and information on all items in each domain including a title, high-level category, description, and URL to a thumbnail image. This dataset also contains various types of feedback signals besides natural language: the set of selected items, survey responses.
Other information:
- **Curated by:** Felix Leeb and Tobias Schnabel
- **Funded by:** Microsoft Research (Augmented Learning and Reasoning team)
- **Language(s) (NLP):** English
- **License:** CDLA Permissive, Version 2.0

### User and Item Features
The dataset broadly contains three types of user feedback over three user-assistant turns:
- **User Profiles**: Structured responses to intake questions on interests, goals, and habits.
- **Behavioral History**: Items selected in each turn
- **Satisfaction**: Ratings of how satisfied or relevant the entire set of recommendations was in each turn
- **Natural Language Messages**: The user's initial request and requested modifications/feedback in each turn

## Uses

The dataset is intended for research on personalized recommendation systems, in particular to study how different types of feedback signals (ratings, written feedback, user profiles) can be used to increase the relevance of recommendations.

## Dataset Structure

The dataset is split into three different files for each of the three domains: news, travel, and food for a total of 9 files. For each domain, there is one file containing all the information about the items that can be recommended `products.csv`, one file containing all the user information `users.csv`, and one file containing the interactions between users and the assistant `impressions.csv` (including the user requests and selections).

The `[domain]-impressions.jsonl` contains three impressions (one for each round) per user:
- `iid`: a unique identifier for the interaction
- `uid`: the user identifier
- `round`: the round number of the interaction (1-3)
- `categories`: a list of categories the user selected for the recommendations
- `request`: the user's request to the assistant
- `selected1`, `selected2`, `selected3`: the user's selection of the recommendations
- `update1`, `update2`: the user's update to the recommendations
- `rating1`, `rating2`, `rating3`: 9-point scale ratings of the recommendations
- `summary`: a free-response summary of the user's feedback on the recommendations
- `rating_summary`: 9-point scale rating of the summary
- `good_suggestions`: a 5-point scale rating of how good the suggestions were
- `good_selections`: a 5-point scale rating of how good the selections were
- `good_request_match`: a 5-point scale rating of how well the recommendations matched the request
- `good_summary_match`: a 5-point scale rating of how well the summary matched the feedback

The `[domain]-products.jsonl` file lists all items in a given domain:
- `pid`: a unique identifier for the item
- `title`: the title of the item (shown to the user)
- `description`: a longer description of the item (shown to the user)
- `thumbnail`: a URL to a thumbnail image of the item (shown to the user)
- `category`: a category label for the item (not shown to the user)

The `[domain]-users.jsonl` contains all static information about each user:
- `uid`: a unique identifier for the user
- `source_types`: a list of categories selected by the user
- `desc_sources`: a free-response description of how the user engages with the domain
- `personalized`: a 5-point scale rating of how much the user values personalized recommendations
- `explore`: a 5-point scale rating of how much the user values exploring new places
- `desc_selection`: a free-response description of the user's selection process
- `task_clear`: a 5-point scale rating of how clear the user found the task
- `task_difficult`: a 5-point scale rating of how difficult the user found the task
- `dialogue_helpful`: a 5-point scale rating of how helpful the user found the dialogue with the assistant
- `recs_satisfied`: a 5-point scale rating of how satisfied the user was with the recommendations
- `task_feedback`: a free-response description of the user's feedback on the task
- Travel-specific fields:
- `companions`: a list of companions the user prefers to travel with
- Food-specific fields:
- `companions`: a list of companions the user eats with or cooks for
- News-specific fields:
- `frequency`: a list of categories the user selected for the recommendations
- `reasons`: a free-response description of why the user reads news
- `demand`: a 5-point scale rating of how much the user would like to use a personalized AI assistant for news

## Dataset Creation

The data was collected using a web-survey with participants recruited through Prolific.
The participants were asked to provide general information about their habits and preferences in one of the three domains, then they were asked to make a specific request to the assistant, and provide feedback on the recommendations they received.
We use a hybrid interface with both free-text and structured input, for example, in item selection, users choose from a visual layout of 12 items with thumbnails.
We choose 12 items without replacement in each round, drawn uniformely at random from the general inventory filtered down to the main interest categories selected in the profile building process in order to ensure both unbiasedness as well as a baseline level of relevance.

### Curation Rationale
We created this dataset because there were no public recommendation datasets that included both conventional feedback signals (ratings, selection) as well as natural language feedback and profile information.

### Dataset Sources
For each domain, we created an inventory:
- News: 629 news articles sourced from [lifehacker.com](https://www.lifehacker.com/)
- Travel: 630 travel destinations sourced from [wikivoyage.org](https://www.wikivoyage.org/)
- Meals: 3904 meals sourced from [blueapron.com](https://www.blueapron.com/cookbook)

## Bias, Risks, and Limitations

Participants in the survey were recruited using Prolific, with the only requirement being that they were fluent in English, and their residence was in the US. This may introduce biases in the dataset, as the participants may not be representative of the general population. Additionally only a small number (approximately 600 participants) were recruited for the survey, which may limit the generalizability of the dataset.

## Citation

**BibTeX:**

```
@misc{recoreact2025arxiv,
title={Do as I say, not just as I do: Understanding Feedback for Recommendation Systems},
author={Felix Leeb and Tobias Schnabel},
year={2025},
eprint={TBD},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={TBD},
}
```

# 概述
RecoReact是一款全新的真实用户与随机分配AI助手（AI assistant）间的多轮交互（multi-turn interactions）数据集，其收录了独特的多类反馈信号集，涵盖多轮自然语言请求、结构化物品选择（structured item selections）、物品评分（item ratings）以及用户画像（user profiles）。该数据集覆盖新闻文章、旅游目的地与膳食规划三大领域。

## 数据集详情
本数据集包含来自595名用户的1785次交互，交互完成时间中位数约为22分钟。RecoReact还包含每位用户的完全匿名化用户画像（user profiles）数据，以及各领域所有物品的相关信息，包括物品标题、高级分类、描述与缩略图URL。除自然语言反馈外，本数据集还包含多种其他反馈信号：用户选定的物品集合、问卷回复。
其他信息：
- **整理方**：Felix Leeb与Tobias Schnabel
- **资助方**：微软研究院（增强学习与推理团队，Microsoft Research (Augmented Learning and Reasoning team)）
- **自然语言处理所用语言**：英语
- **许可证**：CDLA 许可协议2.0版（CDLA Permissive, Version 2.0）

### 用户与物品特征
本数据集围绕三轮用户-助手交互场景，涵盖三类用户反馈：
- **用户画像（user profiles）**：针对兴趣、目标与习惯的结构化初始调研问卷回复
- **行为历史**：每一轮中用户选择的物品
- **满意度**：用户对每一轮所有推荐内容的满意程度与相关程度评分
- **自然语言消息（Natural Language Messages）**：用户向助手发起的初始请求，以及每一轮中提出的修改请求或反馈意见

## 应用场景
本数据集旨在支撑个性化推荐系统相关研究，尤其用于探究如何利用不同类型的反馈信号（评分、书面反馈、用户画像）提升推荐内容的相关性。

## 数据集结构
本数据集针对三大领域分别拆分出三类文件，涵盖新闻、旅游与膳食领域，总计9个文件。每个领域包含：一个存储所有可推荐物品信息的文件（文件名为`products.csv`，实际格式为`[domain]-products.jsonl`）、一个存储所有用户信息的文件（文件名为`users.csv`，实际格式为`[domain]-users.jsonl`），以及一个存储用户与助手交互信息的文件（文件名为`impressions.csv`，实际格式为`[domain]-impressions.jsonl`，包含用户请求与物品选择记录）。

### 各文件详细结构
1. `[domain]-impressions.jsonl`：存储每位用户的三次交互记录（每轮对应一条），字段说明如下：
- `iid`：交互唯一标识符
- `uid`：用户唯一标识符
- `round`：交互轮次编号（1-3）
- `categories`：用户为推荐选定的分类列表
- `request`：用户向助手发起的请求内容
- `selected1`、`selected2`、`selected3`：用户对推荐内容的选择结果
- `update1`、`update2`：用户对推荐内容的更新请求
- `rating1`、`rating2`、`rating3`：采用9分制的推荐内容评分
- `summary`：用户对推荐内容反馈的开放式文本总结
- `rating_summary`：采用9分制的总结评分
- `good_suggestions`：采用5分制的推荐质量评分
- `good_selections`：采用5分制的选择合理性评分
- `good_request_match`：采用5分制的推荐与请求匹配度评分
- `good_summary_match`：采用5分制的总结与反馈匹配度评分

2. `[domain]-products.jsonl`：列出指定领域的所有物品，字段说明如下：
- `pid`：物品唯一标识符
- `title`：向用户展示的物品标题
- `description`：向用户展示的物品详细描述
- `thumbnail`：向用户展示的物品缩略图URL
- `category`：物品的分类标签（不向用户展示）

3. `[domain]-users.jsonl`：存储每位用户的静态信息，字段说明如下：
- `uid`：用户唯一标识符
- `source_types`：用户选定的分类列表
- `desc_sources`：用户参与该领域活动的开放式文本描述
- `personalized`：采用5分制的用户对个性化推荐的重视程度评分
- `explore`：采用5分制的用户对探索新内容的重视程度评分
- `desc_selection`：用户选择流程的开放式文本描述
- `task_clear`：采用5分制的用户对任务清晰程度评分
- `task_difficult`：采用5分制的用户对任务难度评分
- `dialogue_helpful`：采用5分制的用户对与助手对话的有用程度评分
- `recs_satisfied`：采用5分制的用户对推荐内容的满意程度评分
- `task_feedback`：用户对任务的开放式文本反馈描述
- 旅游领域专属字段：
- `companions`：用户偏好的旅行同伴列表
- 膳食领域专属字段：
- `companions`：用户共同用餐或为其烹饪的同伴列表
- 新闻领域专属字段：
- `frequency`：用户选定的新闻阅读频率分类列表
- `reasons`：用户阅读新闻的开放式文本描述
- `demand`：采用5分制的用户对使用新闻个性化AI助手的需求程度评分

## 数据集构建
本数据集通过网络调研收集，调研参与者通过Prolific平台招募。参与者需首先提供其在所选三大领域之一的日常习惯与偏好的基本信息，随后向AI助手发起具体请求，并对收到的推荐内容提供反馈。
我们采用了融合自由文本与结构化输入的混合交互界面：例如在物品选择环节，用户可从包含缩略图的12个物品可视化布局中进行选择。
每一轮我们都会从经用户画像初始流程筛选后的核心兴趣分类的总物品库中，无重复地均匀随机抽取12个物品，以确保数据的无偏性与基础相关性水平。

### 整理初衷
我们创建该数据集的原因是，目前尚无公开的推荐系统数据集能够同时涵盖传统反馈信号（评分、物品选择）、自然语言反馈以及用户画像信息。

### 数据集来源
针对每个领域，我们分别构建了物品库：
- 新闻领域：629篇来自[lifehacker.com](https://www.lifehacker.com/)的新闻文章
- 旅游领域：630个来自[wikivoyage.org](https://www.wikivoyage.org/)的旅游目的地
- 膳食领域：3904道来自[blueapron.com](https://www.blueapron.com/cookbook)的食谱

## 偏差、风险与局限性
本调研的参与者通过Prolific平台招募，仅要求参与者具备英语流利能力且居住在美国。这可能会为数据集引入偏差，因为参与者群体无法代表全体大众。此外，本次调研仅招募了约600名参与者，可能限制了数据集的泛化能力。

## 引用
**BibTeX引用格式：**

@misc{recoreact2025arxiv,
title={Do as I say, not just as I do: Understanding Feedback for Recommendation Systems},
author={Felix Leeb and Tobias Schnabel},
year={2025},
eprint={TBD},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={TBD},
}