rounaksaha12/ai-in-peer-review

---
configs:
- config_name: easy-subset
data_files:
- split: full
path: easy-subset-consolidated.jsonl
- config_name: hard-subset
data_files:
- split: full
path: hard-subset-consolidated.jsonl
- config_name: human-reviews
data_files:
- split: full
path: human-reviews.jsonl
- config_name: post-humanization
data_files:
- split: full
path: post-humanization.jsonl
language:
- en
---

## Overview

This repository contains more than 50,000 reviews of scientific papers, spanning multiple levels of human-AI collaboration.
Following table summarizes these levels:

| Description | Input to LLM |
|---|---|
| AI-BP: AI-generated with Basic Prompts | Paper + Reviewing guidelines
| AI-EP: AI-generated with Elaborate Prompts | Paper + Reviewing guidelines + Conference-issued best practice documents
| AI-HI: AI-generated with Human Input | Paper + Reviewing guidelines + Key assessment points as bulleted list
| H-AI: Human written AI polished | Human written review
| H: Fully Human written review | NA

Fully Human written reviews are from pre-2020 conferences and sourced from the [PeerRead](https://huggingface.co/datasets/allenai/peer_read/blob/main/README.md#dataset-description) dataset

## Files

`easy-subset-consolidated.jsonl` - For each paper, we generate reviews using GPT-4o and Llama-3.3-70B-Instruct models, with one prompt per level. This results in 18,340 AI-generated reviews, which we refer to as the **Easy subset**.

`hard-subset-consolidated.json` - For a subset of $158$ papers ($18$ from CoNLL 2016 and $20$ from each of the remaining conferences), we generate reviews using GPT-5, Gemini-2.5-pro, Gemma-3-27b-it, Qwen-3-30B-thinking and Llama-3.1-70B-instruct. For this subset, we employ at least 4 distinct prompts per level, resulting in 27,429 AI-generated reviews, which we refer to as the **Hard subset**.

`human-reviews.jsonl` - This contains the 3499 human-written reviews

`post-humanization.jsonl` - Humanization refers to the paraphrasing of AI-generated text, with the deliberate aim of mimicking human writing to evade detection. We employ [Undetectable AI](https://undetectable.ai/) one of the most widely-used commercial AI humanizers with an API to humanize reviews from two key levels: AI-BP and H-AI. There are 2000 such reviews.

## Data fields

Each jsonl file contains the following fields:

`conference` - name and year of the conference

`level` - one of the five levels of human intervention: AI-BP, AI-EP, AI-HI, H-AI and H. Humanized reviews marked with level of human intervention of source review followed by the suffix '-humanized'

`paper_number` - unique identifier of the paper given the conference name and year

`reviewer_id` - unique identifier of the reviewer given the `conference` and `paper_number`

`generating_model` - model used to generate the review. For fully human written reviews (H), this entry is `human_review`

`prompt_id` - unique id of the prompt used to generate the review, formatted as `[level]@[prompt_number]`. For fully human written reviews (H), this entry is `HUMAN`

`review_text` - the contents of the review

`filepath` - unique identifying string for the review

`split` - train/test/dev