huggingworld/BRIGHT

BRIGHT基准是首个需要密集推理的文本检索基准，查询来自多个真实领域（如StackExchange、LeetCode和数学竞赛），基于人类实际数据。实验表明，现有检索模型在BRIGHT上表现不佳，最高nDCG@10得分仅为22.1。该基准为未来在更现实和挑战性环境下的检索研究提供了良好测试平台。数据集包含examples、documents和long_documents三个子集，涵盖生物学、地球科学、经济学、心理学、机器人学、StackOverflow、可持续生活、LeetCode、Pony、AoPS和TheoremQA等多个主题，支持多种模型配置（如Gemini-1.0、Claude-3-Opus、GPT-4等）。

BRIGHT is the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. The queries are collected from diverse domains (StackExchange, LeetCode, and math competitions), all sourced from realistic human data. Experiments show that existing retrieval models perform poorly on BRIGHT, where the highest score is only 22.1 measured by nDCG@10. BRIGHT provides a good testbed for future retrieval research in more realistic and challenging settings. The dataset includes three subsets: examples, documents, and long_documents, covering topics such as biology, earth science, economics, psychology, robotics, StackOverflow, sustainable living, LeetCode, Pony, AoPS, and TheoremQA, with multiple model configurations (e.g., Gemini-1.0, Claude-3-Opus, GPT-4).