mesolitica/Malaysian-Text-Benchmark

这是一个轻量级的基准数据集，用于评估马来西亚语境下的合成生成。它包括各种语言特定的数据集，用于翻译任务，以及对不同语言和方言的可靠性测试。这些数据集由人类进行注释或偏好选择，以进行更好的评估。数据包括马来语、英语、印尼语、泰米尔语、普通话、爪哇语、Manglish、登嘉楼、吉打和登嘉楼方言的示例。文件还解释了每个数据集的目的和用法，并以JSON和Python格式提供了示例数据。

This is a lightweight benchmark dataset for evaluating synthetic generation in the Malaysian context. It includes various language-specific datasets for translation tasks, and reliability tests for different languages and dialects. The datasets are annotated or preferred by humans for better evaluation. The data includes examples in Malay, English, Indonesian, Tamil, Mandarin, Jawi, Manglish, Negeri Sembilan, Kedah, and Kelantan dialects. The file also explains the purpose and usage of each dataset, and provides sample data in JSON and Python formats.