Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals (data)

Data accompanying the paper: Vladislav Poritski, Oksana Volchek, Maksim Aparovich, Volha Harytskaya, Pavel Smrž. Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals. Proceedings of LoResLM 2026, p. 378–387. There are two folders in the archive data.zip: tasks and completions. In tasks, each subdirectory represents one Belarusian-language task in OpenAI Evals library. The subdirectory names follow the naming conventions of the library, i.e. they are more detailed than the task names in the article but the correspondence is easy to follow, for example: belarusian_syllable_count = Syllable count, belarusian_numerals = Numerals, etc. Each subdirectory contains two files: dev.jsonl has been copied from the library, and test.jsonl has been created from scratch. In completions, there are four files: {comm,open}_{dev,test}.csv. These are human-readable outputs of each commercial or open-weight model evaluated against the dev and test sets of each task. The columns are: model ID, temperature (None for o4-mini; 0.0 to 1.0 with step size 0.2 for Gemma and Qwen; 0.0 or 1.0 otherwise), task name, fold name (dev or test), instance index in the fold (zero-based), the user prompt of the instance, expected answer (newline-separated if several distinct answers are accepted), the model's completion, and correctness (1 or 0) automatically checked by OpenAI Evals.