JetBrains-Research/nes-mixed-v9-memorization

该数据集包含多个子集，用于评估代码生成模型在给定不同数量前/后示例（k）时的表现。每个样本包括示例索引、前k数量、后k数量、显示的代码片段、真实代码、模型响应、训练数据标识、反事实响应，以及部分子集中模型响应与真实代码的距离、反事实响应与真实代码的距离。数据集可用于分析模型在代码补全任务中的鲁棒性和反事实推理能力。This dataset contains multiple subsets for evaluating code generation models under varying numbers of before/after examples (k). Each sample includes an example index, before_k, after_k, shown code snippet, ground truth, model response, trained_on indicator, contrafactual response, and in some subsets, distances between response and ground truth and between contrafactual response and ground truth. It is designed for analyzing model robustness and counterfactual reasoning in code completion tasks.