minishlab/tokenlearn-cornstack-docs-coderankembed-v2

该数据集是使用Tokenlearn工具创建的，旨在为代码检索任务训练Model2Vec模型。它包含了由nomic-ai/CodeRankEmbed模型生成的均值token嵌入，这些嵌入作为静态嵌入蒸馏的训练目标。数据集来源于CornStack，覆盖了六种编程语言：Python、Java、PHP、Go、JavaScript和Ruby，每种语言提供100,000个代码文档，总计600,000行。每个数据样本包括两个字段：text（截断的输入文本，最大长度为512个token）和embedding（来自CodeRankEmbed的均值token嵌入列表，维度为768，排除了BOS和EOS令牌）。数据集可用于多语言代码表示学习，支持加载单个语言配置或合并所有语言。创建过程使用了两个采样种子（42和100）以及10k流式洗牌缓冲区以最大化多样性。This dataset was created with Tokenlearn for training Model2Vec models on code retrieval. It contains mean token embeddings produced by nomic-ai/CodeRankEmbed, used as training targets for static embedding distillation. The dataset includes code documents from CornStack across six programming languages: Python, Java, PHP, Go, JavaScript, and Ruby, with 100,000 rows per language and 600,000 total rows. Each sample consists of two fields: text (truncated input text with a maximum length of 512 tokens) and embedding (a list of mean token embeddings from CodeRankEmbed, with a dimension of 768, excluding BOS and EOS tokens). It is designed for multilingual code representation learning and supports loading individual language configurations or concatenating all languages. The creation process involved two sampling seeds (42 and 100) and a 10k streaming shuffle buffer to maximize diversity.