UQABench

[KDD'25] UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering [KDD 2025 Accepted (Oral) Paper]
## Overview
The paper link: [UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering.](https://arxiv.org/abs/2502.19178)

Github: https://github.com/OpenStellarTeam/UQABench

The source data (Kaggle): [Kaggle](https://www.kaggle.com/datasets/liulangmingliu/uqabench)

## Description
The UQABench is a benchmark dataset for evaluating user embeddings in prompting LLMs for personalized question answering. The standardized evaluation process includes **pre-training**, **fine-tuning**, and **evaluating** stages. We provide the requirements and quick-start scripts for each stage.

The source data are user interactions collected and processed from Taobao. Following previous work, we randomly split the data into 9:1 as the training and test sets. The dataset statistics are summarized as follows:

| Data Split | Total | #Training | #Test |
|---------------|-------------|------------|------------|
| Interaction | 31,317,087 | 28,094,799 | 3,222,288 |

Specifically, the training set serves in the pre-training and fine-tuning (aligning) stages. Then, we design task-specific question prompts based on the test set. We refine the questions, filter out low-quality questions, and eventually get 7,192 personalized Q&A for the evaluating stage.

## Citation
Please cite our paper if you use our dataset.
```
@inproceedings{liu2025uqabench,
title={UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering},
author={Liu, Langming and Liu, Shilei and Yuan, Yujin and Zhang, Yizhen and Yan, Bencheng and Zeng, Zhiyuan and Wang, Zihao and Liu, Jiaqi and Wang, Di and Su, Wenbo and others},
booktitle={Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2},
pages={5652--5661},
year={2025}
}
```

[KDD'25] UQABench：评估用于个性化问答中提示大语言模型(LLM)的用户嵌入 [KDD 2025 录用（口头报告）论文]
## 概览
论文链接：[UQABench：评估用于个性化问答中提示大语言模型的用户嵌入](https://arxiv.org/abs/2502.19178)
Github仓库：https://github.com/OpenStellarTeam/UQABench
源数据集（Kaggle平台）：[Kaggle](https://www.kaggle.com/datasets/liulangmingliu/uqabench)

## 数据集说明
UQABench是一款用于评估个性化问答场景下提示大语言模型所用用户嵌入的基准数据集。其标准化评估流程涵盖**预训练**、**微调**与**评估**三个阶段，我们为每个阶段提供了需求说明与快速启动脚本。

该数据集的源数据源自淘宝平台收集并处理后的用户交互记录。遵循过往研究范式，我们将数据以9:1的比例随机划分为训练集与测试集。数据集统计信息如下：

| 数据划分 | 总样本量 | 训练集样本数 | 测试集样本数 |
|---------------|-------------|------------|------------|
| 交互记录 | 31,317,087 | 28,094,799 | 3,222,288 |

具体而言，训练集将用于预训练与微调（对齐）阶段。随后，我们基于测试集设计了面向特定任务的问答提示词。经过问题打磨与低质量样本过滤后，最终得到7192条用于评估阶段的个性化问答对。

## 引用说明
若您使用本数据集，请引用我们的论文：

@inproceedings{liu2025uqabench,
title={UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering},
author={Liu, Langming and Liu, Shilei and Yuan, Yujin and Zhang, Yizhen and Yan, Bencheng and Zeng, Zhiyuan and Wang, Zihao and Liu, Jiaqi and Wang, Di and Su, Wenbo and others},
booktitle={Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2},
pages={5652--5661},
year={2025}
}