AI and evaluating research quality

Assessing
the quality of studies is crucial for meta-analyses, but this is a
time-consuming and labor-intensive procedure. While recent advances in Large
Language Models (LLMs) offer a potential solution, their ability to provide
reliable appraisals remains largely unclear. In this study, we compared how two
humans and two LLMs rated the quality of forty educational intervention studies
based on a standardized tool. The agreement on the overall quality rating
ranges from none to fair across the four raters (-.02 ≤ κ ≤ .38), where the
human-human agreement was the highest. However, agreement levels with respect
to different quality criteria varied dramatically, and in several cases, the
agreements among the LLMs surpassed the human-human agreement. The
disagreements stemmed from missing information in the papers, legitimate
diverging considerations, and raters’ errors. These findings highlight both the
promise and limitations of using LLMs to evaluate the quality of educational
research.