On Inter-dataset Code Duplication and Data Leakage in Large Language Models

This dataset encompasses the sparse graph referenced in the publication titled "On Inter-dataset Code Duplication and Data Leakage in Large Language Models." This resource is a snapshot of the original repository, and the graph is preserved in the interduplication.db database. The schema of this database is easily understandable and is available in the original repository. Each code snippet is identified by a unique identifier (id_within_dataset) that corresponds to its identification within the dataset from which it was extracted. The complete datasets are stored in .jsonl files within their respective folders (e.g., python-150/data.jsonl, codetrans/data.jsonl, etc.).本数据集包含发表于论文《论大语言模型中数据集间代码重复与数据泄露》（On Inter-dataset Code Duplication and Data Leakage in Large Language Models）中所引用的稀疏图（sparse graph）。 本资源为原始代码仓库的快照，该稀疏图存储于interduplication.db数据库中。该数据库的模式（schema）清晰易懂，相关说明文档可在原始代码仓库中获取。每个代码片段均配有唯一标识符（id_within_dataset），该标识符与其所提取自的原始数据集内的标识一一对应。完整数据集以.jsonl文件格式存储于各自对应的文件夹中（例如python-150/data.jsonl、codetrans/data.jsonl等）。