rapidata-vs-prolific

# Rapidata vs. Prolific: the same 300 questions on both platforms In July 2026 we asked the crowds of two annotation platforms — [Prolific](https://www.prolific.com) and [Rapidata](https://www.rapidata.ai) — the same 300 two-image questions and graded every response against known correct answers. This dataset is the complete raw material of that comparison: the questions, the images, and all 18,007 individual responses from both crowds. Each question is a short instruction and two images, one of which is correct. 92 questions are simple object recognition ("Select the image that shows a Snake."); 208 target one of six attention skills: counting, absence, search, spatial relations, object state, and attribute binding. Nothing requires special knowledge — a question is only ever hard because it takes a careful look.  ## How the two crowds compare | | Prolific | Rapidata | | -------------------- | ----------- | ---------- | | People | 150 | 5,103 | | Answers per question | 10 | 50 | | Total answers | 3,000 | 15,007 | | Collection time | ~66 minutes | ~2 minutes |  Graded one answer at a time, Prolific's crowd is more accurate: 98.0% against Rapidata's 91.6%. In practice, labels are made by majority vote over several answers per question, and there the two crowds converge:  Prolific's majority labels all 300 questions correctly from seven responses per question. Rapidata reaches 99.9% at nine responses and labels all 300 correctly from 23. At full depth, neither crowd got a single question majority-wrong. Rapidata's answers cost $4 per 1,000 against Prolific's $76, so the larger majorities it needs barely register in the price of a finished label:  ## Schema One row per question. | column | description | |---|---| | `prompt` | The instruction shown to annotators, e.g. "Select the image that shows exactly five yellow rubber ducks." | | `image1`, `image2` | The two images of the pair, embedded (display order was randomized per annotator on both platforms) | | `correct_image` | The ground truth: which image matches the prompt (`image1`/`image2`) | | `category` | `object-recognition` or one of `attention-counting`, `attention-absence`, `attention-search`, `attention-spatial`, `attention-state`, `attention-binding` | | `difficulty` | Measured difficulty class (`easy` / `medium`) | | `task_id` | Task identifier (`t001`-`t300`) | | `votes_image1_rapidata`, `votes_image2_rapidata` | Rapidata vote counts per image (~50 per question) | | `weighted_results_image1_rapidata`, `weighted_results_image2_rapidata` | Rapidata vote shares weighted by each annotator's chance-corrected score (sum to 1) | | `detailed_results_rapidata` | JSON list of every Rapidata response: `votedFor` (`image1`/`image2`), `annotatorId`, `userScore`, `country`, `language`, `sessionIndex`, `createdAt` | | `votes_image1_prolific`, `votes_image2_prolific` | Prolific vote counts per image (10 per question) | | `detailed_results_prolific` | JSON list of every Prolific response: `votedFor`, `annotatorId`, `responseTimeMs`, `createdAt` | ```python from datasets import load_dataset import json ds = load_dataset("Rapidata/rapidata-vs-prolific", split="train") # overall single-answer accuracy, per platform for platform in ["rapidata", "prolific"]: votes = [ v["votedFor"] == r["correct_image"] for r in ds for v in json.loads(r[f"detailed_results_{platform}"]) ] print(platform, sum(votes) / len(votes)) # rapidata 0.916, prolific 0.980 ``` ## Notes on the data Annotator identifiers are anonymized sequential ids (`r0001`…, `p0001`…). They preserve which answers came from the same person but cannot be linked back to platform accounts. Prolific participants were paid a fixed reward sized from the estimated completion time at the platform-recommended £19.40/hour. Rapidata responses were collected with a single API order at standard pricing, no audience filtering. The attention-skill images were generated with gpt-image-2.