cross-encoder/lightonai-embeddings-fine-tuning-reranked-v1

该数据集是LightOn embeddings-fine-tuning的重新评分版本，使用mxbai-rerank-large-v2模型对原始查询-候选文档对进行评分，旨在作为知识蒸馏的教师数据，用于训练重排模型学生。数据集包含多个配置：documents（文档ID和文本）、queries（查询ID和文本）、scores（每个查询与2048个候选文档的评分，包括正例ID和分数）、scores_merged（合并所有正例的评分）、scores_subsampled（子采样到256文档的评分）和scores_merged_subsampled（合并正例并子采样到256文档的评分）。数据按多个检索领域（如fiqa、hotpotqa、msmarco、nq、fever、squadv2、trivia）分割，支持文本排序和检索任务。子采样逻辑包括强制包含正例、选择前16个最难负例和基于分位数锚点的分层采样，以保留教师模型的分数分布。该数据集适用于点对MSE蒸馏和列表对蒸馏训练，是ettin-reranker-v1模型系列的上游数据源。

This dataset is a teacher-rescored version of LightOn embeddings-fine-tuning, using the mxbai-rerank-large-v2 model to score original query-candidate document pairs, intended as teacher data for knowledge distillation to train reranker students. It includes multiple configs: documents (document IDs and texts), queries (query IDs and texts), scores (scores for each query with 2048 candidate documents, including positive IDs and scores), scores_merged (scores with all positives merged), scores_subsampled (subsampled to 256 documents per query), and scores_merged_subsampled (merged positives and subsampled to 256 documents per query). The data is split by retrieval domains (e.g., fiqa, hotpotqa, msmarco, nq, fever, squadv2, trivia) and supports text ranking and retrieval tasks. The subsampling logic involves forcing inclusion of positives, selecting the top 16 hardest negatives, and quantile-anchor stratified sampling to preserve the teachers score distribution. The dataset is suitable for pointwise MSE distillation and listwise/pairwise distillation training, serving as the upstream artifact for the ettin-reranker-v1 model family.