LLM-Generated Software Requirements from GitHub Issues

This dataset provides a collection of software requirements automatically generated by Large Language Models (LLMs) based on feature requests extracted from five prominent open-source repositories on GitHub: pytorch/pytorch, flutter/flutter, godotengine/godot-proposals, rust-lang/rust, and golang/go. The requirements were generated using two distinct LLMs: o3-mini (via OpenAI API) and deepseek-r1-distill-llama-70b (via Groq API). Three different prompting strategies were employed for generation: a simple zero-shot (naive) prompt, a prompt assigning an expert persona, and a few-shot prompt providing examples. The generated requirements were subsequently evaluated for quality using the qwen-qwq-32b model (via Groq API). The evaluation focused on three standard requirement quality criteria: Unambiguity, Verifiability, and Singularity, with scores assigned on a 1–5 scale along with a textual explanation. The dataset is structured to facilitate analysis of requirement quality based on the generating LLM and the prompting technique used. Dataset Contents issues.csvContains the source feature requests, including the repository name, issue number, and issue title, extracted from the target GitHub repositories. all_requirements.csvStores the software requirements generated by the o3-mini and deepseek-r1-distill-llama-70b models for each issue. Includes the source issue details, the generating LLM, the prompt style used (zero_shot, few_shot, expert), and the generated requirement text. parsed.csvContains the quality assessment results for each generated requirement. Includes the 1–5 scores for Unambiguity, Verifiability, and Singularity, along with a textual explanation for the scores provided by the qwen-qwq-32b model. Links back to the original issue and the specific generated requirement (LLM and prompt style). Assets (Visualizations) boxplot_deepseek.pngBoxplot visualizing the distribution of evaluation scores for requirements generated by the deepseek-r1-distill-llama-70b model, broken down by prompt style. boxplot_o3-mini.pngBoxplot visualizing the distribution of evaluation scores for requirements generated by the o3-mini model, broken down by prompt style. requirements_evaluation_histogram.pngHistograms comparing the overall score distribution for each evaluation metric (Unambiguity, Verifiability, Singularity) between the o3-mini and deepseek-r1-distill-llama-70b models. Scripts (Code Used for Generation and Analysis) prompts.pyDefines the LangChain prompt templates used for requirement generation, including zero-shot, expert persona, and few-shot examples. extract_issues.pyScript utilizing the PyGithub library to fetch open feature request issues from the specified GitHub repositories based on labels and sorting criteria. generate_requirements.pyOrchestrates the requirement generation process. It iterates through the extracted issues, applies the different prompt styles, invokes the specified LLMs (o3-mini, deepseek), processes the responses, and saves the generated requirements and metadata. evaluate_requirements.pyPerforms the evaluation of the generated requirements. Uses the qwen-qwq-32b LLM via LangChain with a structured output Pydantic model (Evaluation) to assess each requirement against the defined criteria (Unambiguity, Verifiability, Singularity) and saves the results. analyze_evaluations.ipynbA Jupyter Notebook containing Python code (using Pandas, Matplotlib, Seaborn, SciPy) to load the evaluation data, perform descriptive statistical analysis, conduct hypothesis testing (Mann-Whitney U, Kruskal-Wallis) to compare LLMs and prompt styles, and generate the visualizations included in the assets.