bigcode/github-commits-diff-dedup-pjjs-april
收藏Hugging Face2023-04-17 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/bigcode/github-commits-diff-dedup-pjjs-april
下载链接
链接失效反馈官方服务：
资源简介：
# Deduplicated Commits
Deduplicated based on diff:
```
content = '\n'.join(difflib.unified_diff(
old_content.splitlines(keepends=True),
new_content.splitlines(keepends=True),
n=5
))
```
## Parameters:
Minimum ngram size: 5
MinHash ngram size: 5
MinHash threshold: 0.8
提供机构：
bigcode原始信息汇总
Deduplicated Commits
数据集处理方法
- 去重基于差异（diff）： python content = .join(difflib.unified_diff( old_content.splitlines(keepends=True), new_content.splitlines(keepends=True), n=5 ))
参数设置
- 最小ngram大小：5
- MinHash ngram大小：5
- MinHash阈值：0.8
