MarcoDotIO/jpn-bench

JPN-Bench是一个用于分词器评估和未来日语大语言模型评估的日语识字基准测试。此公开版本包含一个经过精心策划的小型分词器识字开发集，以及基准测试源材料清单，这些材料与分词器训练材料分开保存。该数据集与KotodamaLM分词器工作一起分组在Hugging Face集合“KotodamaLM Japanese Language Infrastructure”中。JPN-Bench is a Japanese literacy benchmark for tokenizer evaluation and future Japanese LLM evaluation. This public release contains a small curated tokenizer-literacy dev set plus benchmark-lane source material manifests kept separate from tokenizer training material. This dataset is grouped with the KotodamaLM tokenizer work in the Hugging Face collection KotodamaLM Japanese Language Infrastructure.