How do LLMs Answer Open Coding Questions?

This artifact contains the dataset used in the paper How do LLMs Answer Open Coding Questions?. The study investigates how large language models answer open Stack Overflow questions that do not have accepted answers, and how those answers are judged by other LLMs, manual inspection, and Stack Overflow community feedback. The dataset contains 280 open Stack Overflow questions from seven programming domains: JavaScript, Python, Java, C#, PHP, Android, and HTML. Four LLMs were used as answer generators: gpt-4o, claude-3-7-sonnet-20250219, qwen3-235b-a22b-thinking-2507, and gemini 2.5 pro. Each model answered all 280 questions, resulting in 1120 generated answers. Each generated answer was reviewed by the other three LLMs, yielding 3360 peer-review judgement files. The artifact also includes RQ2 machine relevance judgements, manual relevance labels for 280 sampled answers, and Stack Overflow feedback evidence for submitted answers. Package structure: paper_data_release/ README.md data/ raw_generation_peer_review/ <generator>/<domain>/<question_id>/ prompt.txt original_answer.txt peer_review/ *_vote.txt automated_verdict.txt manual_check.txt submitted_answer.txt relevance_prompt.txt relevance_results/ *_relevant.txt RELEVANT.txt or NOT_RELEVANT.txt feedback/ *.png similar_answer_exists/ tables/ questions_metadata.csv rq1_peer_review_summary.csv rq2_integrated_machine_manual.csv rq2_manual_relevance_labels.csv rq2_metrics_by_domain.csv rq2_metrics_by_model.csv rq2_kappa_by_domain.csv rq2_kappa_by_model.csv rq3_community_feedback.csv rq4_vote_rates_by_generator.csv rq4_feedback_by_generator.csv scripts/ audit_release.py compute_rq2_metrics.py Main contents: questions_metadata.csv: metadata for the 280 selected Stack Overflow questions, including vote score, view count, question age, and asker reputation. raw_generation_peer_review/: per-answer raw data. Each folder contains the original prompt, generated answer, peer-review judgement files, RQ2 relevance prompt and relevance results, and optional manual/submission/feedback files. rq1_peer_review_summary.csv: answer-level peer-review summary for all 1120 generated answers. rq2_integrated_machine_manual.csv: answer-level table integrating peer-review results and RQ2 machine relevance/plausibility judgements. rq2_manual_relevance_labels.csv: 280 manually inspected relevance labels in English-coded form. rq2_metrics_by_domain.csv, rq2_metrics_by_model.csv, rq2_kappa_by_domain.csv, and rq2_kappa_by_model.csv: manual-vs-machine metric summaries. rq3_community_feedback.csv: cleaned row-level Stack Overflow feedback/candidate records. This table has 188 valid rows: 153 relevant submitted answers and 35 not-relevant candidate records. rq4_vote_rates_by_generator.csv: LLM peer-review vote-rate summary by answer generator and reviewer. rq4_feedback_by_generator.csv: paper-level community feedback summary for the 153 submitted answers, including deleted, negative, positive/neutral, no-feedback, and accepted outcomes. Consistency with the paper: 280 selected open Stack Overflow questions. 7 programming domains. 4 answer-generating LLMs. 1120 generated answers. 3360 LLM peer-review judgement files. 280 valid manual relevance labels. RQ2 overall relevance F1 score: 0.732530, reported as approximately 0.733 in the paper. RQ3 paper-level feedback summary: 153 submitted answers, 39 deleted answers, 23 negative-feedback answers, 20 positive/neutral-feedback answers, 71 no-feedback answers, and 2 accepted answers. RQ4 generator vote rates match the paper-level summary: Gemini 85.2%, Qwen 86.5%, GPT-4o 38.0%, and Claude 55.6%. Reuse: To verify the release package: cd paper_data_release python3 scripts/audit_release.py To recompute RQ2 manual-vs-machine relevance metrics from the released raw folders: cd paper_data_release python3 scripts/compute_rq2_metrics.py本附属材料包含论文《大语言模型如何解答开放式编码问题？》（How do LLMs Answer Open Coding Questions?）中所使用的数据集。本研究探究了大语言模型（Large Language Model，LLM）如何解答尚无已采纳回答的开放式Stack Overflow问题，以及这些回答如何通过其他LLM、人工审核以及Stack Overflow社区反馈进行评判。 本数据集包含来自7个编程领域的280个开放式Stack Overflow问题，涉及领域包括JavaScript、Python、Java、C#、PHP、Android及HTML。研究选用了4款LLM作为回答生成器：gpt-4o、claude-3-7-sonnet-20250219、qwen3-235b-a22b-thinking-2507及gemini 2.5 pro。每款模型均对全部280个问题进行解答，共计生成1120条回答。每条生成的回答均由其余3款LLM进行评审，共计产生3360份同行评审判断文件。本附属材料还包含RQ2的机器相关性判断结果、280个抽样回答的人工相关性标注，以及提交至Stack Overflow的回答的社区反馈证据。 包目录结构： paper_data_release/ README.md data/ raw_generation_peer_review/ <generator>/<domain>/<question_id>/ prompt.txt original_answer.txt peer_review/ *_vote.txt automated_verdict.txt manual_check.txt submitted_answer.txt relevance_prompt.txt relevance_results/ *_relevant.txt RELEVANT.txt 或 NOT_RELEVANT.txt feedback/ *.png similar_answer_exists/ tables/ questions_metadata.csv rq1_peer_review_summary.csv rq2_integrated_machine_manual.csv rq2_manual_relevance_labels.csv rq2_metrics_by_domain.csv rq2_metrics_by_model.csv rq2_kappa_by_domain.csv rq2_kappa_by_model.csv rq3_community_feedback.csv rq4_vote_rates_by_generator.csv rq4_feedback_by_generator.csv scripts/ audit_release.py compute_rq2_metrics.py 核心内容： questions_metadata.csv：280个精选Stack Overflow问题的元数据文件，涵盖投票得分、浏览量、问题发布时长以及提问者信誉等信息。 raw_generation_peer_review/：单条回答的原始数据目录。每个子文件夹均包含原始提示词、生成的回答、同行评审判断文件、RQ2相关性提示词与相关性结果，以及可选的人工审核、提交回答与反馈相关文件。 rq1_peer_review_summary.csv：全部1120条生成回答的回答级同行评审汇总数据。 rq2_integrated_machine_manual.csv：整合了同行评审结果与RQ2机器相关性/合理性判断的回答级数据表。 rq2_manual_relevance_labels.csv：以英文编码形式呈现的280份人工相关性标注结果。 rq2_metrics_by_domain.csv、rq2_metrics_by_model.csv、rq2_kappa_by_domain.csv及rq2_kappa_by_model.csv：人工与机器指标的汇总结果文件。 rq3_community_feedback.csv：清洗后的行级Stack Overflow社区反馈/候选记录数据表，共包含188条有效记录：153条相关提交回答与35条不相关候选记录。 rq4_vote_rates_by_generator.csv：按回答生成器与评审者分组的LLM同行评审投票率汇总数据。 rq4_feedback_by_generator.csv：153条提交回答的论文级社区反馈汇总结果，涵盖已删除、负面反馈、正面/中性反馈、无反馈以及已采纳等多种结果。 与论文的一致性： 1. 280个精选开放式Stack Overflow问题 2. 7个编程领域 3. 4款回答生成LLM 4. 1120条生成回答 5. 3360份LLM同行评审判断文件 6. 280份有效人工相关性标注 7. RQ2整体相关性F1值：0.732530，论文中报告为约0.733 8. RQ3论文级反馈汇总：153条提交回答、39条已删除回答、23条负面反馈回答、20条正面/中性反馈回答、71条无反馈回答以及2条已采纳回答 9. RQ4生成器投票率与论文级汇总结果一致：Gemini 85.2%、Qwen 86.5%、GPT-4o 38.0%及Claude 55.6% 复用方式： 如需验证发布包： cd paper_data_release python3 scripts/audit_release.py 如需从发布的原始文件夹中重新计算RQ2的人工与机器相关性指标： cd paper_data_release python3 scripts/compute_rq2_metrics.py