search-arena-24k

## Overview

This dataset contains **ALL** in-the-wild conversation crowdsourced from [Search Arena](https://lmarena.ai/?search) between March 18, 2025 and May 8, 2025. It includes **24,069** multi-turn conversations with search-LLMs across diverse intents, languages, and topics—alongside **12,652** human preference votes. The dataset spans approximately 11,000 users across 136 countries, 13 publicly released models, around 90 languages (including 11\% multilingual prompts), and over 5,000 multi-turn sessions.

While user interaction patterns with general-purpose LLMs and traditional search engines are increasingly well-understood, we believe that search-LLMs represent a new and understudied interface—blending open-ended generation with real-time retrieval. This hybrid interaction mode introduces new user behaviors: how questions are posed, how retrieved information is interpreted, and what types of responses are preferred. We release this dataset to support analysis of this emerging paradigm in human–AI interaction, grounded in large-scale, real-world usage.

All users consented the terms of use to share their conversational data with us for research purposes, and all entries have been redacted using Google’s Data Loss Prevention (DLP) API to remove PII and sensitive information.

Each data point includes:
- Two model standardized responses (`messages_a` and `messages_b`)
- The human vote result (half of them have this feature)
- Timestamp
- Full system metadata, LLM + web search trace
- Post-processed annotations such as language and user intent

## Transparency Statement
We clarify the following for transparency:
(1) No pre-release models were deployed to Search Arena during data collection.
(2) All model responses are anonymized to ensure that user preferences reflect conversational quality rather than brand identity.
(3) All anonymized data is publicly released; no model provider received privileged access.

For more details, see our [blog post](https://blog.lmarena.ai/blog/2025/search-arena).

## License
User prompts are licensed under CC-BY-4.0, and model outputs are governed by the terms of use set by the respective model providers.

## Citation
```bibtex
@misc{searcharena2025,
title = {Introducing the Search Arena: Evaluating Search-Enabled AI},
url = {https://blog.lmarena.ai/blog/2025/search-arena/},
author = {Mihran Miroyan*, Tsung-Han Wu*, Logan Kenneth King, Tianle Li, Anastasios N. Angelopoulos, Wei-Lin Chiang, Narges Norouzi, Joseph E. Gonzalez},
month = {April},
year = {2025}
}

@misc{chiang2024chatbot,
title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference},
author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica},
year={2024},
eprint={2403.04132},
archivePrefix={arXiv},
primaryClass={cs.AI}
}
```

## 数据集概览

本数据集包含**全部**于2025年3月18日至2025年5月8日期间，从[搜索竞技场（Search Arena）](https://lmarena.ai/?search)众包采集的真实场景对话数据。其中包含**24069**轮涉及搜索-大语言模型（search-LLMs）的多轮对话，覆盖多样的用户意图、语言与主题，同时附带**12652**条人类偏好投票。本数据集覆盖136个国家的约11000名用户、13个公开发布的模型、近90种语言（其中11%为多语言提示词），以及超过5000个多轮交互会话。

尽管用户与通用大语言模型及传统搜索引擎的交互模式已逐渐被学界明晰，但搜索-大语言模型作为融合开放式生成与实时检索的新型交互界面，仍未得到充分研究。这种混合交互模式催生了全新的用户行为：包括问题的提问方式、检索信息的解读逻辑，以及偏好的响应类型。本数据集旨在支撑针对这一新兴人机交互范式的分析研究，其数据均来自大规模真实使用场景。

所有用户均已签署使用条款，同意共享其对话数据用于科研用途，且所有条目均已通过谷歌数据丢失防护（Data Loss Prevention, DLP）API进行脱敏处理，以移除个人可识别信息（Personally Identifiable Information, PII）及敏感信息。

每个数据样本包含以下内容：
- 两份经过模型标准化的响应（`messages_a`与`messages_b`）
- 人类投票结果（其中半数样本具备该特征）
- 时间戳
- 完整系统元数据、大语言模型+网页检索追踪记录
- 后处理标注信息，如语言类别与用户意图

## 透明度声明

为确保透明度，我们特此说明以下事项：
(1) 数据采集期间，搜索竞技场未部署任何预发布模型。
(2) 所有模型响应均已匿名化处理，以确保用户偏好反映的是对话质量而非品牌辨识度。
(3) 所有匿名化数据均已公开发布，未向任何模型提供商开放特权访问权限。

更多细节可参阅我们的[博客文章](https://blog.lmarena.ai/blog/2025/search-arena)。

## 许可协议

用户提示语采用CC-BY-4.0协议授权，模型输出需遵循各模型提供商的使用条款。

## 引用格式
bibtex
@misc{searcharena2025,
title = {Introducing the Search Arena: Evaluating Search-Enabled AI},
url = {https://blog.lmarena.ai/blog/2025/search-arena/},
author = {Mihran Miroyan*, Tsung-Han Wu*, Logan Kenneth King, Tianle Li, Anastasios N. Angelopoulos, Wei-Lin Chiang, Narges Norouzi, Joseph E. Gonzalez},
month = {April},
year = {2025}
}

@misc{chiang2024chatbot,
title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference},
author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica},
year={2024},
eprint={2403.04132},
archivePrefix={arXiv},
primaryClass={cs.AI}
}