LLM-Generated Software Requirements from GitHub Issues

This dataset provides a collection of software requirements automatically generated by Large Language Models (LLMs) based on feature requests extracted from five prominent open-source repositories on GitHub: pytorch/pytorch, flutter/flutter, godotengine/godot-proposals, rust-lang/rust, and golang/go. The requirements were generated using two distinct LLMs: o3-mini (via OpenAI API) and deepseek-r1-distill-llama-70b (via Groq API), across three prompting strategies: zero-shot (naive), expert persona, and few-shot. The generated requirements were subsequently evaluated for quality using the qwen-qwq-32b model on three ISO/IEC/IEEE 29148:2018 criteria: Unambiguity, Verifiability, and Singularity (1–5 Likert scale). What is new in v2.0.0 This version extends the replication package with data and scripts from the Human Validation Study conducted in the second revision of the associated paper. Five human evaluators with backgrounds in Software Engineering and Requirements Engineering independently rated a stratified random sample of 50 requirements on the same three quality criteria, blinded to LLM identity, prompt strategy, and LLM-judge scores. Five agreement metrics were computed (Krippendorff's alpha, Spearman's rho, Cohen's weighted kappa, exact/within-±1 agreement, and signed mean deviation). All evaluator identities have been anonymized (E1–E5). Additionally, all figures have been regenerated with English axis labels, shared Y-axis scales, and larger fonts for improved readability. Dataset Contents Core Dataset (unchanged from v1) issues.csv — Source feature requests (repository, issue number, issue title). all_requirements.csv — 900 generated requirements (LLM × prompt style × issue). parsed.csv — LLM-judge quality scores and rationales for all 900 requirements. Human Validation Study (new in v2) human_validation__human_validation_sample.csv — Stratified random sample of 50 requirements selected for human evaluation, covering all LLM × prompt style × score-bin combinations. human_validation__human_validation_answer_key.csv — LLM-judge scores and metadata for the 50 sampled requirements. human_validation__evaluators__E1.csv – E5.csv — Individual ratings from each of the five anonymized human evaluators. human_validation__results__summary.csv — Aggregated agreement metrics across all criteria. human_validation__results__krippendorff_alpha.csv — Inter-rater agreement among human evaluators. human_validation__results__spearman_correlation.csv — Spearman's rho between LLM-judge scores and aggregate human judgment. human_validation__results__cohen_kappa.csv — Cohen's weighted kappa per evaluator vs. LLM judge. human_validation__results__percentage_agreement.csv — Exact and within-±1 agreement rates. human_validation__results__signed_mean_deviation.csv — Mean signed deviation between LLM-judge and human scores (bias direction). Figures fig_1.png — Study methodology overview. fig_2.png — Score distribution histograms by LLM (English labels, shared Y-axis). fig_3.png — Boxplots by prompt style for o3-mini (regenerated with larger fonts). fig_4.png — Boxplots by prompt style for DeepSeek R1 (regenerated with larger fonts). boxplot_o3-mini.png, boxplot_deepseek.png, requirements_evaluation_histogram.png — Original v1 figures retained for reference. Scripts prompts.py — LangChain prompt templates (zero-shot, expert, few-shot). extract_issues.py — GitHub issue extraction via PyGithub. generate_requirements.py — Requirement generation pipeline. evaluate_requirements.py — LLM-as-a-Judge evaluation pipeline. analyze_evaluations.ipynb — Statistical analysis and visualization notebook. scripts__stratified_sample.py — Stratified sampling of 50 requirements for human validation (new in v2). scripts__convert_validation_results.py — Converts raw evaluator responses to a unified CSV format (new in v2). scripts__compute_agreement_metrics.py — Computes all five agreement metrics (new in v2). scripts__regenerate_figures.py — Regenerates figures with English labels and shared axes (new in v2).