Are Large Language Models Reliable Argument Quality Annotators?

This is a dataset of 320 arguments, each annotated with 15 different argument quality dimensions. The annotations were performed by 2 groups of human annotators: expert and novice, as well as large language models with different prompt variations.  

Please find more details in the corresponding publication: https://webis.de/publications.html#mirzakhmedova_2024b

Code for experiments can be found at: https://github.com/webis-de/RATIO-24

Please use the following citation key: 

@InProceedings{mirzakhmedova:2024b,  author =                   {Nailia Mirzakhmedova and Marcel Gohsen and Chia Hao Chang and Benno Stein},  booktitle =                {1st International Conference on Recent Advances in Robust Argumentation Machines {(RATIO-24)}},  doi =                      {10.1007/978-3-031-63536-6_8},  editor =                   {Philipp Cimiano and Anette Frank and Michael Kohlhase and Benno Stein},  month =                    jun,  pages =                    {129--146},  publisher =                {Springer},  site =                     {Bielefeld, Germany},  title =                    {{Are Large Language Models Reliable Argument Quality Annotators?}},  volume =                   14638,  year =                     2024}