DL-HARD

DL-HARD数据集是由格拉斯哥大学和马克斯普朗克信息学研究所共同创建，旨在更有效地评估神经排名模型在复杂主题上的表现。该数据集基于TREC Deep Learning（DL）主题，通过广泛标注问题意图类别、答案类型、维基化实体、主题类别和商业网络搜索引擎的结果类型元数据来构建。DL-HARD包含从2019年和2020年的官方DL评估基准中选取的50个最具挑战性的主题，其中25个是新评估的。数据集的应用领域主要集中在识别和解决神经排名模型在处理复杂和挑战性主题时的不足，推动神经排名方法的研究进展。

DL-HARD Dataset was co-developed by the University of Glasgow and the Max Planck Institute for Informatics, with the goal of more effectively evaluating the performance of neural ranking models on complex topics. Built upon TREC Deep Learning (DL) topics, this dataset is constructed through extensive annotation of question intent categories, answer types, wikified entities, topic categories, and result type metadata from commercial web search engines. DL-HARD includes the 50 most challenging topics selected from the official DL evaluation benchmarks of 2019 and 2020, 25 of which are newly evaluated. The primary applications of this dataset lie in identifying and resolving the shortcomings of neural ranking models when handling complex and challenging topics, so as to promote the advancement of research in neural ranking methods.