CMiLBench: A Hierarchical Multitask Benchmark for Low-Resource Minority Languages in China

As large language models (LLMs) continue to advance, they achieve strong performance on high-resource language tasks and show promising potential for low-resource language processing. However, existing benchmarks primarily focus on high-resource languages, with limited coverage of Chinese minority languages. To address this gap, we introduce CMiLBench (Chinese Minority Language Benchmark), a comprehensive evaluation framework targeting three key Chinese minority languages: Tibetan, Mongolian, and Uyghur. CMiLBench comprises 17 task categories and 24,663 samples, covering both language understanding and generation. Several tasks are derived from native corpora and culturally grounded content, offering a realistic assessment of model performance in authentic minority language scenarios. Tasks are stratified into five difficulty levels, and evaluation is conducted using both automatic metrics and LLM-as-a-Judge scoring. We evaluate 14 leading commercial and open-source LLMs, demonstrating that CMiLBench serves as a reliable benchmark and is broadly applicable for evaluating LLMs’ capabilities in Chinese minority languages, thereby calibrating current technological progress and advancing research and application development of multilingual models for low-resource languages.