Linguistic-Diagnostics-Syntax

# LINDSEA Syntax LINDSEA Syntax is a linguistic diagnostic from [BHASA](https://arxiv.org/abs/2309.06085) that evaluates a model's understanding of linguistic phenomena, syntax in particular, for Indonesian. ### Supported Tasks and Leaderboards LINDSEA Syntax is designed for evaluating chat or instruction-tuned large language models (LLMs). ### Languages - Indonesian (id) ### Dataset Details LINDSEA Syntax only has an Indonesian (id) split, with additional splits containing fewshot examples. Below are the statistics for this dataset. The number of tokens only refer to the strings of text found within the `prompts` column. | Split | # of examples | # of GPT-4o tokens | # of Gemma 2 tokens | # of Llama 3 tokens | |-|:-|:-|:-|:-| | id | 380 | 9493 | 9273 | 11293 | | id_fewshot | 5 | 112 | 116 | 130 | | **total** | 385 | 9605 | 9389 | 11423 | ### Data Sources | Data Source | License | Language/s | Split/s |-|:-|:-| :-| | [BHASA](https://arxiv.org/abs/2309.06085) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Indonesian | id, id_fewshot ### License For the license/s of the dataset/s, please refer to the data sources table above. We endeavor to ensure data used is permissible and have chosen datasets from creators who have processes to exclude copyrighted or disputed data. ## Acknowledgement This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative. ### References ```bibtex @misc{leong2023bhasaholisticsoutheastasian, title={BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models}, author={Wei Qi Leong and Jian Gang Ngui and Yosephine Susanto and Hamsawardhini Rengarajan and Kengatharaiyer Sarveswaran and William Chandra Tjhi}, year={2023}, eprint={2309.06085}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2309.06085}, } ```