How do LLMs Answer Open Coding Questions?

This artifact contains the dataset used in the paper How do LLMs Answer Open Coding Questions?. The study investigates how large language models answer open Stack Overflow questions that do not have accepted answers, and how those answers are judged by other LLMs, manual inspection, and Stack Overflow community feedback. The dataset contains 280 open Stack Overflow questions from seven programming domains: JavaScript, Python, Java, C#, PHP, Android, and HTML. Four LLMs were used as answer generators: gpt-4o, claude-3-7-sonnet-20250219, qwen3-235b-a22b-thinking-2507, and gemini 2.5 pro. Each model answered all 280 questions, resulting in 1120 generated answers. Each generated answer was reviewed by the other three LLMs, yielding 3360 peer-review judgement files. The artifact also includes RQ2 machine relevance judgements, manual relevance labels for 280 sampled answers, and Stack Overflow feedback evidence for submitted answers. Package structure: paper_data_release/ README.md data/ raw_generation_peer_review/ <generator>/<domain>/<question_id>/ prompt.txt original_answer.txt peer_review/ *_vote.txt automated_verdict.txt manual_check.txt submitted_answer.txt relevance_prompt.txt relevance_results/ *_relevant.txt RELEVANT.txt or NOT_RELEVANT.txt feedback/ *.png similar_answer_exists/ tables/ questions_metadata.csv rq1_peer_review_summary.csv rq2_integrated_machine_manual.csv rq2_manual_relevance_labels.csv rq2_metrics_by_domain.csv rq2_metrics_by_model.csv rq2_kappa_by_domain.csv rq2_kappa_by_model.csv rq3_community_feedback.csv rq4_vote_rates_by_generator.csv rq4_feedback_by_generator.csv scripts/ audit_release.py compute_rq2_metrics.py Main contents: questions_metadata.csv: metadata for the 280 selected Stack Overflow questions, including vote score, view count, question age, and asker reputation. raw_generation_peer_review/: per-answer raw data. Each folder contains the original prompt, generated answer, peer-review judgement files, RQ2 relevance prompt and relevance results, and optional manual/submission/feedback files. rq1_peer_review_summary.csv: answer-level peer-review summary for all 1120 generated answers. rq2_integrated_machine_manual.csv: answer-level table integrating peer-review results and RQ2 machine relevance/plausibility judgements. rq2_manual_relevance_labels.csv: 280 manually inspected relevance labels in English-coded form. rq2_metrics_by_domain.csv, rq2_metrics_by_model.csv, rq2_kappa_by_domain.csv, and rq2_kappa_by_model.csv: manual-vs-machine metric summaries. rq3_community_feedback.csv: cleaned row-level Stack Overflow feedback/candidate records. This table has 188 valid rows: 153 relevant submitted answers and 35 not-relevant candidate records. rq4_vote_rates_by_generator.csv: LLM peer-review vote-rate summary by answer generator and reviewer. rq4_feedback_by_generator.csv: paper-level community feedback summary for the 153 submitted answers, including deleted, negative, positive/neutral, no-feedback, and accepted outcomes. Consistency with the paper: 280 selected open Stack Overflow questions. 7 programming domains. 4 answer-generating LLMs. 1120 generated answers. 3360 LLM peer-review judgement files. 280 valid manual relevance labels. RQ2 overall relevance F1 score: 0.732530, reported as approximately 0.733 in the paper. RQ3 paper-level feedback summary: 153 submitted answers, 39 deleted answers, 23 negative-feedback answers, 20 positive/neutral-feedback answers, 71 no-feedback answers, and 2 accepted answers. RQ4 generator vote rates match the paper-level summary: Gemini 85.2%, Qwen 86.5%, GPT-4o 38.0%, and Claude 55.6%. Reuse: To verify the release package: cd paper_data_release python3 scripts/audit_release.py To recompute RQ2 manual-vs-machine relevance metrics from the released raw folders: cd paper_data_release python3 scripts/compute_rq2_metrics.py