lance-format/ms-marco-v2.1-lance

MS MARCO v2.1 QA（Lance格式）是一个基于MS MARCO v2.1的Lance格式版本，源自Microsoft的机器阅读理解基准，构建自匿名Bing查询日志。每行数据包含一个用户查询、Bing检索到的多达10个候选段落（带有相关性标志）、人工编写的参考答案（包括标准答案和完整句子形式答案），以及预计算的384维MiniLM查询嵌入（使用sentence-transformers/all-MiniLM-L6-v2模型，余弦归一化）。数据集采用列式存储，支持快速随机访问和高效查询，内置IVF_PQ向量索引、INVERTED全文搜索索引、BTREE标识符索引和BITMAP查询类型索引。它分为训练集（808,731行）和验证集（101,093行），适用于问答、文本检索、信息检索等任务，特别适合RAG（检索增强生成）、答案评估和机器学习训练工作流。数据集支持多种加载方式，包括HuggingFace datasets、LanceDB和Lance库，并具备数据演化、版本控制和子集材料化功能。MS MARCO v2.1 QA (Lance Format) is a Lance-formatted variant based on MS MARCO v2.1, derived from Microsoft's machine reading comprehension benchmark and constructed from anonymous Bing query logs. Each row contains a user query, up to 10 candidate passages retrieved by Bing with relevance flags, manually curated reference answers (including standard answers and full-sentence formatted responses), as well as pre-computed 384-dimensional MiniLM query embeddings generated using the sentence-transformers/all-MiniLM-L6-v2 model with cosine normalization. The dataset adopts columnar storage, supporting fast random access and efficient querying, and is equipped with built-in IVF_PQ vector indexes, INVERTED full-text search indexes, BTREE identifier indexes and BITMAP query type indexes. It is split into a training set (808,731 rows) and a validation set (101,093 rows), applicable to tasks including question answering, text retrieval and information retrieval, and particularly suitable for RAG (Retrieval-Augmented Generation), answer evaluation and machine learning training workflows. The dataset supports multiple loading methods including HuggingFace Datasets, LanceDB and the Lance library, and provides capabilities such as data evolution, version control and subset materialization.