lmarena-ai/arena-human-preference-55k

--- license: apache-2.0 task_categories: - text-classification language: - en pretty_name: LMSYS Chatbot Arena Human Preference Predictions size_categories: - 10K<n<100K --- Dataset for [Kaggle competition](https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview) on predicting human preference on Chatbot Arena battles. The training dataset includes over 55,000 real-world user and LLM conversations and user preferences across over 70 state-of-the-art LLMs, such as GPT-4, Claude 2, Llama 2, Gemini, and Mistral models. Each sample represents a battle consisting of 2 LLMs which answer the same question, with a user label of either prefer model A, prefer model B, tie, or tie (both bad). ### Citation Please cite the following paper if you find our leaderboard or dataset helpful. ``` @misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} } ```许可证：Apache-2.0 任务类别： - 文本分类 语言： - 英语 友好名称：LMSYS 聊天机器人竞技场人类偏好预测数据集 样本规模区间： - 10000 < 样本数 < 100000 本数据集为对应Kaggle竞赛（链接：https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview）的官方数据集，任务为预测聊天机器人竞技场对决中的人类偏好选择。 训练集包含超过5.5万条真实场景下的用户与大语言模型（Large Language Model，LLM）对话数据，以及覆盖70余种当前前沿大语言模型的用户偏好标注，涉及GPT-4、Claude 2、Llama 2、Gemini、Mistral等多款主流模型。 每条样本对应一场模型对决：两个大语言模型针对同一提问生成应答，用户标注标签分为「偏好模型A」「偏好模型B」「平局」以及「平局（两者均不佳）」四类。 ### 引用 若您认为本竞赛排行榜或数据集对您的研究有所帮助，请引用以下论文： @misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }