Webis Crowd RAG Corpus 2025

Data Documentation

responses.jsonl.gz

RAG responses of about 250 words written by human writers and LLMs in different response styles. 

Key
Description
Source
Values

response
The UUID of the response / task.
Task Specification
UUID

topic
The topic ID of this task.
Task Specification
String ID

style
The text style of the response written for this task.
Task Specification
One of essay, news, bullet

kind
Whether this text was written by an LLM or a human
Task Specification
One of human, llm

query
The query text of this topic.
TREC RAG
String value

references_ids
The IDs of the 20 sources retrieved for this topics' query. Aligned with references_texts
TREC RAG
List of String IDs

references_texts
The texts of the 20 sources retrieved for this topics' query. Aligned with references_ids
TREC RAG
List of String values

text
The text as written by the human author or LLM.
Writing Survey
String

cleaned_text
Text as cleaned by our preprocessing pipeline, without reference markers.
Writing Survey
String

statements
Text parsed into individual statements, each with the corresponding references_ids cited.
Writing Survey
List of Dictionaries

ratings.jsonl.gz

Ratings on pairwise response utility as given by crowd workers. The columns prefixed {dimension} below are included once for each possible dimension (correctness_topical, coherence_logical, coherence_stylistic, coverage_broad, coverage_deep, consistency_internal, quality_overall).

Key
Description
Source
Value

submission_id
The UUID of the questionnaire this response pair was rated by.
Task Specification
UUID

query_id
The topic id this response pair belongs to.
TREC RAG
String ID

response_a
The UUID of the first response in this pair (displayed lefthand side).
Task Specification
UUID

response_b
The UUID of the second response in this pair (displayed righthand side).
Task Specification
UUID

worker
The UUIDs of the 5 workers completing this questionnaire.
Task Specification
List of UUID

{dimension}_vote
The individual votes for the specified dimension by the 5 workers.
Prolific Crowd Workers
List of string, each entry a, n, or b

{dimension}_spam_probability
The individual spam probabilities associated with each vote for the specified dimension.
Prolific Crowd Workers
List of float, each entry between 0 and 1

{dimension}_p_a
The probability of the gold label being a for the specified dimension (first response better than second).
Prolific Crowd Workers
float

{dimension}_p_n
The probability of the gold label being n for the specified dimension (both responses equal).
Prolific Crowd Workers
float

{dimension}_p_b
The probability of the gold label being b for the specified dimension (second response better than first).
Prolific Crowd Workers
float

{dimension}_gold
The gold label with highest probability for the specified dimension.
Prolific Crowd Workers
a, n, or b

llm_ratings.jsonl.gz

Ratings on pairwise response utility as given by an LLM. The columns prefixed {dimension} below are included once for each possible dimension (correctness_topical, coherence_logical, coherence_stylistic, coverage_broad, coverage_deep, consistency_internal, quality_overall).

Key
Description
Source
Value

submission_id
The UUID of the questionnaire this response pair was rated by.
Task Specification
UUID

query_id
The topic id this response pair belongs to.
TREC RAG
String ID

response_a
The UUID of the first response in this pair (displayed lefthand side).
Task Specification
UUID

response_b
The UUID of the second response in this pair (displayed righthand side).
Task Specification
UUID

inference
The inference mode the judgments were collected with.
Task Specification
combined, or individual

{dimension}
The rating given by the LLM for this {dimension}.
LLM Inference
a, n, or b

grades.jsonl.gz

Pointwise, per-topic ranked grades as inferred by a Bradley-Terry probabilistic model. Not to be used as absolute values across their topic context!

Key
Description
Source
Value

response
The UUID of the response.
Task Specification
UUID

correctness_topical
The topical correctness grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

coherence_logical
The logical coherence grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

coherence_stylistic
The stylistic coherence grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

coverage_broad
The broad coverage grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

coverage_deep
The deep coverage grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

consistency_internal
The internal consistency grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.

quality_overall
The overall quality grade of this response.
Pairwise Inference w. Bradley-Terry Model
Integer, 1-6, per topic relative ranks, higher is better.