Data and code for "Benchmark Reuse and Endpoint Drift in LLM-Based Vulnerability Detection: An Empirical Study of Four Chinese APIs"
收藏资源简介:
Dataset and analysis code for the manuscript of the same title. The study evaluates four Chinese LLM APIs on Java vulnerability detection with the JavaVulBench benchmark, in two collection rounds: 3,200 calls on 2026-07-27 (four APIs, three prompt strategies, two data splits) and 4,600 calls between 2026-10-03 and 2026-10-06 (grid re-run, same-period direct-verdict panel, a fifth vendor, a successor endpoint, a third snapshot, and an 800-call GPT-4o replication). The record contains the archived raw responses and the corrected per-record verdicts, the official 200-sample probes, the fine-tuned encoder predictions, the training/evaluation overlap audit, and the code that regenerates every table and figure: the verdict parser with its acceptance test, the metric and bootstrap routines, the table and figure generators, the encoder training script, the non-LLM baseline, and code/reproduce.sh, which runs the CPU-side pipeline end to end. Table 5 of the manuscript is reproduced with three rows; the kimi-k3 run, collected at the provider-mandated temperature 1, is reproduced separately and reported in the manuscript's Appendix C.



