minishlab/tokenlearn-cornstack-queries-coderankembed-v2

该数据集由Tokenlearn创建，用于在代码检索任务上训练Model2Vec模型。它包含来自CornStack的代码文档，涵盖六种编程语言（Python、Java、PHP、Go、JavaScript、Ruby），每种语言有10万行数据，总计60万行。每个数据样本包括两个字段：text（截断的输入文本，最大标记长度为512）和embedding（由nomic-ai/CodeRankEmbed模型生成的均值标记嵌入向量，维度为768，排除了BOS/EOS标记）。数据集用于静态嵌入蒸馏的训练目标，通过两个采样种子（42和100）和10k流式洗牌缓冲区最大化多样性，文本被截断为512个标记。This dataset was created with Tokenlearn for training Model2Vec models on code retrieval. It contains mean token embeddings produced by nomic-ai/CodeRankEmbed, used as training targets for static embedding distillation. The dataset contains code documents from CornStack across 6 programming languages (100,000 rows per language, 600,000 total). Each sample includes text (truncated input text with tokenizer max length 512) and embedding (mean token embedding from nomic-ai/CodeRankEmbed, excluding BOS/EOS tokens, dimension 768). Two sampling seeds (42 and 100) were used with a 10k streaming shuffle buffer to maximise diversity, and texts are truncated to 512 tokens.