CoReEval: A Large-Scale Dataset for Evaluating Code Readability with Human and Machine Judgments

Dataset Description CoReEval (Code Readability Evaluation) is a large-scale evaluation dataset designed to compare code readability judgments produced by humans, Large Language Models (LLMs), and a traditional deep learning model (Scalabrino et al.).The dataset was created as part of the study presented in the anonymous paper:“From Metrics to Models: Large Language Models as Human-Aligned Code Readability Evaluators.” Contents The dataset is provided in three complementary formats: File Name Format Approx. Size readability_results.csv CSV 789 MB readability_results.json JSON 1.44 GB readability_results_part*.xlsx (x9) Excel ~23 MB per file Purpose This dataset enables in-depth analysis of how well LLMs can replicate human judgments of code readability. It covers: Multiple programming languages: Java, Python, and CUDA Two code types: Production code (code) and unit tests (test) Three established open-source datasets: Buse, Dorn, and Scalabrino Two developer profiles: Junior and Senior Two prompt families: templates (generic prompts) templates-2 (developer-guided, human-centric prompts) Dataset Structure Each row in the dataset represents the readability evaluation of a code snippet by a specific model or method. The fields include: Field Description group One of templates, templates-2, or scalabrino_direct persona JUNIOR or SENIOR (for LLM-generated results) dataset Dataset origin: Buse, Dorn, or Scalabrino language Programming language: java, python, or cuda code_type code (production code) or test (unit test) model_name Name of the LLM or SCALABRINO (for the deep learning baseline) prompt_type Prompt strategy: ZSL, FSL, CoT, ToT, or None (for Scalabrino) top_p Decoding parameter top_p (LLMs only) temperature Decoding temperature (LLMs only) iteration Iteration number (1 to 3) snippet_id Unique identifier of the evaluated snippet score Raw score (1–20) predicted by the model normalized_score Normalized score (1–20) likert_category Readability category: readable, neutral, or unreadable (Likert 1–5) groundtruth_score Corresponding human Likert score reasoning Justification text (LLMs only; None for Scalabrino baseline) Large Language Models Used The dataset includes results from a wide range of API-based and locally hosted Large Language Models (LLMs), covering diverse architectures and providers. Model Provider Size Access Claude 3 Sonnet Anthropic Unknown API Deepseek Chat Deepseek ~64B API GPT-4o OpenAI Unknown API Mistral Large (v2407) Mistral ~65B (Mixture-of-Experts) API Deepseek LLM 7B Chat Deepseek (HF) 7B Local Falcon 3 7B Instruct TII (HF) 7B Local Gemma 7B IT Google (HF) 7B Local Mistral 7B Instruct Mistral (HF) 7B Local Phi-3 Mini Microsoft (HF) 4.2B Local Qwen2.5 7B Instruct Alibaba (HF) 7B Local Decoding Parameter Grid Each LLM was evaluated using a full grid of decoding parameters to assess generation variability and robustness. Temperature ↓ / Top_p → 0.5 0.75 1.0 0.1 Yes Yes Yes 0.5 Yes Yes Yes 1.0 Yes Yes Yes This results in 9 decoding settings per prompt, per model, per snippet, and per iteration. All combinations were used consistently to ensure coverage of deterministic to more creative generation settings. Replicability Each configuration (model_name, prompt_type, top_p, temperature) was run across three different iterations to ensure robustness and capture variability.In total, the dataset contains over 1.4 million readability judgments. Anonymity All information related to the authors, institutions, or affiliations has been removed to comply with double-blind reviewing requirements. Usage Scenarios This dataset can be used to: Evaluate the ability of LLMs to replicate human readability judgments Study the impact of prompt engineering on model outputs Analyze correlations between different models, profiles, and settings Explore the structure and quality of reasoning generated by LLMs