Tokenizer for VietNamese
收藏kaggle2018-03-16 更新2024-03-07 收录
下载链接：
https://www.kaggle.com/datasets/luanho/tokenizervietnamese
下载链接
链接失效反馈官方服务：
资源简介：
应用场景：
创建时间：
2018-03-16
相关数据集
nbdaaa/vietnamese_curated_tokenized_dataset
--- dataset_info: features: - name: input_ids list: int32 - name: attention_mask list: int8 splits: - name: train num_bytes: 51062776945 num_examples: 10000000 download_siz
Hugging Face2026-01-09 更新50
The Tokenization Tax: Token Fertility and Production Impact Data for Vietnamese LLMs
Experimental dataset accompanying the paper 'The Tokenization Tax: Measuring Production Impact of Token Over-Consumption in Open-Source LLMs for Vietnamese-Language Workloads'. Contains raw token fert
Zenodo2026-05-23 更新10
nhantruongcse/summary-vietnamese-news-token-TFeval_vit5_base
--- dataset_info: features: - name: Content dtype: string - name: Summary dtype: string - name: input_ids sequence: int32 - name: attention_mask sequence: int8 - name: labe
Hugging Face2023-12-08 更新120
The Tokenization Tax: Token Fertility and Production Impact Data for Vietnamese LLMs
Experimental dataset accompanying the paper 'The Tokenization Tax: Measuring Production Impact of Token Over-Consumption in Open-Source LLMs for Vietnamese-Language Workloads'. Contains raw token fert
Zenodo2026-05-23 更新00
trungbb8/vietnamese-news-corpus-tokenized
--- license: apache-2.0 configs: - config_name: default data_files: - split: train path: data/train-* dataset_info: features: - name: input_ids list: int32 splits: - na
Hugging Face2026-04-04 更新20
