bigcode/santacoder-token-usage
收藏Hugging Face2023-02-17 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/bigcode/santacoder-token-usage
下载链接
链接失效反馈官方服务：
资源简介：
---
dataset_info:
features:
- name: token
dtype: int64
- name: Java
dtype: int64
- name: JavaScript
dtype: int64
- name: Python
dtype: int64
splits:
- name: train
num_bytes: 1571808
num_examples: 49119
download_size: 1165252
dataset_size: 1571808
---
# Dataset Card for "santacoder-token-usage"
Token usage count per language when tokenizing the `"bigcode/stack-dedup-alt-comments"` dataset with the `santacoder` tokenizer. There are less tokens than in the tokenizer because of vocabulary mismatch between the datasets used to train the tokenizer and the ones that ended up being used to train the model.
[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
提供机构：
bigcode
原始信息汇总
数据集概述
数据集名称
santacoder-token-usage
数据集特征
- token: int64
- Java: int64
- JavaScript: int64
- Python: int64
数据集分割
- train:
- 样本数量: 49119
- 数据大小: 1571808 字节
数据集大小
- 下载大小: 1165252 字节
- 数据集大小: 1571808 字节
