rahulurmaliya/cornstack-python-v1

CoRNStack数据集是一个大规模高质量的训练数据集，专门用于跨多种编程语言的代码检索。该数据集包含用于训练nomic-embed-code、CodeRankEmbed和CodeRankLLM的<查询，正例，负例>三元组。数据集基于去重后的Stackv2构建，从函数文档字符串和相应代码创建文本-代码对，过滤了非英语、过短、包含URL、HTML标签或无效字符的低质量对，并保留文档字符串长度在256个标记或以上的对以帮助模型学习长距离依赖。通过双重一致性过滤去除潜在噪声示例，并使用基于课程的硬负采样策略确保模型从具有挑战性的示例中学习。

The CoRNStack Dataset is a large-scale high quality training dataset specifically for code retrieval across multiple programming languages. This dataset comprises of <query, positive, negative> triplets used to train nomic-embed-code, CodeRankEmbed, and CodeRankLLM. It is curated from deduplicated Stackv2 by creating text-code pairs from function docstrings and respective code, with filtering for quality (e.g., English docstrings, minimum length, no URLs/HTML/invalid characters), dual-consistency filtering to remove noisy examples, and employs a curriculum-based hard negative mining strategy during training.