nasa-impact/nasa-sde-IR-benchmark-20251024-v5
收藏资源简介:
NASA SDE IR基准测试v5是一个全面的信息检索基准数据集,专为NASA科学发现引擎(SDE)设计。该数据集包含合成生成的查询-文档对,用于科学内容检索评估。它是NASA SDE IR基准测试v3的更新版本,基于新的SDE内容生成,这些内容未包含在之前的训练数据或基准测试中。数据集包括56,310个初始数据点,从中采样了9,977个数据点用于对生成,最终选择了9,241个高质量数据点。数据集提供两种类型的查询-文档对:QA对(问题-答案对,附带提取的上下文)和搜索对(搜索词-文档对)。这些对使用GPT-4o mini生成,每个文档生成5-15对。数据集结构包括corpus.jsonl(文档集合)、queries.jsonl(查询集合)和qrels文件夹(相关性判断文件,如qa_pairs.tsv和search_pairs.tsv)。数据生成过程涉及源数据筛选、基于内容相关性置信度得分选择前10%、使用GPT-4o mini生成对,并进行质量控制以确保科学准确性和相关性。该基准测试适用于评估信息检索系统,特别是专注于科学内容的系统,可与BEIR等标准IR评估框架一起使用。评估指标包括检索指标(如Recall@k、Precision@k、MAP、MRR、nDCG@k)、问答指标(如精确匹配、F1分数)和搜索指标(如命中率、成功率)。
A comprehensive Information Retrieval benchmark dataset for the NASA Science Discovery Engine (SDE), containing synthetically generated query-document pairs for scientific content retrieval evaluation. This dataset is an updated version of the NASA SDE IR Benchmark v3, generated from new SDE content that wasnt included in previous training data or benchmarks. It includes 56,310 initial data points, with 9,977 datapoints sampled for pair generation and 9,241 high-quality data points selected. The dataset features two types of query-document pairs: QA Pairs (question-answer pairs with extracted context) and Search Pairs (search term-document pairs), generated using GPT-4o mini with 5-15 pairs per document. The dataset structure consists of corpus.jsonl (document collection), queries.jsonl (query collection), and qrels files (relevance judgments, such as qa_pairs.tsv and search_pairs.tsv). The data generation process involves filtering source data, selecting the top 10% based on content relevancy confidence scores, generating pairs with GPT-4o mini, and applying quality control for scientific accuracy and relevance. This benchmark is designed for evaluating Information Retrieval systems, particularly those focused on scientific content, and can be used with standard IR evaluation frameworks like BEIR. Evaluation metrics include retrieval metrics (e.g., Recall@k, Precision@k, MAP, MRR, nDCG@k), question answering metrics (e.g., Exact Match, F1 Score), and search metrics (e.g., Hit Rate, Success Rate).




