arena-human-preference-55k

Dataset for [Kaggle competition](https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview) on predicting human preference on Chatbot Arena battles. The training dataset includes over 55,000 real-world user and LLM conversations and user preferences across over 70 state-of-the-art LLMs, such as GPT-4, Claude 2, Llama 2, Gemini, and Mistral models. Each sample represents a battle consisting of 2 LLMs which answer the same question, with a user label of either prefer model A, prefer model B, tie, or tie (both bad). ### Citation Please cite the following paper if you find our leaderboard or dataset helpful. ``` @misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} } ```本数据集用于[Kaggle竞赛](https://www.kaggle.com/competitions/lmsys-chatbot-arena/overview)，任务为预测Chatbot Arena对决中的人类偏好。 训练数据集包含超过5.5万条真实场景下用户与大语言模型（LLM）的对话数据，以及覆盖70余款当前顶尖大语言模型的用户偏好标注，涉及GPT-4、Claude 2、Llama 2、Gemini及Mistral等系列模型。 每条样本对应一场对决：两款大语言模型针对同一问题给出回答，用户标注的偏好标签分为偏好模型A、偏好模型B、平局，以及平局（两者均不佳）四类。 ### 引用 若您认为本排行榜或数据集对研究有所助益，请引用以下论文： @misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }

搜集汇总

背景与挑战

背景概述

该数据集包含55,000多条真实用户与多种先进LLM的对话及用户偏好数据，用于预测Chatbot Arena中的人类偏好对战结果，覆盖GPT-4、Claude 2等70多种模型。