minishlab/tokenlearn-cornstack-docs-coderankembed

该数据集是使用Tokenlearn工具创建的，用于训练Model2Vec模型进行代码检索。数据集包含了来自CornStack的代码文档，覆盖6种编程语言（Python、Java、PHP、Go、JavaScript、Ruby），每种语言有50,000行数据，总计300,000行。数据集中包含了由nomic-ai/CodeRankEmbed生成的均值令牌嵌入，用作静态嵌入蒸馏的训练目标。数据集的结构包括文本列和嵌入列，文本列是截断的输入文本（最大长度为512个令牌），嵌入列是来自nomic-ai/CodeRankEmbed的均值令牌嵌入（不包括BOS/EOS令牌）。

This dataset was created with Tokenlearn for training Model2Vec models on code retrieval. It contains mean token embeddings produced by nomic-ai/CodeRankEmbed, used as training targets for static embedding distillation. The dataset contains code documents from CornStack across 6 programming languages (50,000 rows per language, 300,000 total). The dataset structure includes a text column (truncated input text with a max length of 512 tokens) and an embedding column (mean token embeddings from nomic-ai/CodeRankEmbed, excluding BOS/EOS tokens).