To Trust or Not to Trust: Authors' Response to AI-based Reviews

Background: Large language models are increasingly discussed and used as tools that may assist with scholarly peer review, but empirical evidence regarding how authors use and perceive AI-based feedback remains limited.Methods: This paper reports findings from two independent pilot studies on authors’ use and perceptions of AI-based auxiliary review at two computer science venues. After review decisions were released, authors were invited to complete an anonymous post-review questionnaire about the AI review’s usefulness, trustworthiness, agreement with human reviews, practical value for revision, perceived inaccuracies, and consent. The final dataset included 56 analyzable responses from authors of 40 papers; closed-ended items were summarized using descriptive statistics, and open-ended responses were analyzed using inductive thematic analysis.Results: Most respondents (83.9%) considered the AI-based review useful, and 80.4% reported that it identified issues not mentioned by human reviewers. This perceived added value translated into action: 82.1% reported using at least some AI feedback in their camera-ready version. However, the authors did not treat the AI review as equivalent to a human review. They generally trusted it less than the human reviews and found human feedback clearer, even though 25.0% described at least some human reviews as not very useful. Reported problems with the AI review were usually limited: 51.8% reported minor inaccuracies, while 16.1% reported clearly incorrect, misleading, or irrelevant comments. Support for future use was strongest when AI was framed as a supervised or author-controlled tool: 96.4% said they would use AI as an internal review tool before future submissions, 89.3% preferred advance notice that AI would be used in review, and 76.8% favored explicit consent before use.Conclusions: In two pilot studies, authors often perceived a clearly labeled AI-based review as a useful supplementary source of feedback, but not as a substitute for human review. The findings support a cautious auxiliary role for AI-based review, with particular attention to prompt design, human oversight, transparency, and consent.