AI-Assisted Systematic Literature Reviews: To Trust or not to Trust? A Preliminary Study

This repository provides additional data to accompany the paper entitled "AI-Assisted Systematic Literature Reviews: To Trust or not to Trust? A Preliminary Study".

The main goal of the paper is to investigate the feasibility of utilizing AI tools, specifically OpenAI GPT-4 Turbo, Google Gemini 2.5 Pro and Microsoft Copilot, to automate the data extraction phase in Systematic Literature Reviews (SLRs).

The repository contains the following files:

- Evaluation_form.xlsx: Form used to compare the semantic similarity of responses to SLR questions generated by different generative AI models with those generated manually. The evaluators had to rate the semantic similarity on a scale from 1 to 5, where 1 indicates that the responses have completely different meanings and 5 indicates that they represent exactly the same meaning. In order to do so, they had to enter the ratings into the grey cells. The file contains a first sheet with 5 research questions (RQs) and 5 additional questions (AQs), as well as 17 sheets containing the responses to each question for 17 primary studies (PS).

- Answers_summaries.xlsx: This file contains the semantic similarity ratings given by different evaluators, as well as summaries of the results classified by the generative AI models used. It also contains model comparisons and Intraclass Correlation Coefficient (ICC) calculations to validate the set of results.