CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers

54 Python snippets covering nine cryptographic misuse classes (36 vulnerable, 18 matched secure controls), a self-contained harness that queries local Ollama models with a fixed verdict-only prompt, and 1,890 recorded trials across seven open code models (qwen2.5-coder 0.5B to 14B, deepseek-coder 6.7B, codellama 7B). Includes an analysis script that reproduces per-model and per-class detection rates with Wilson 95% confidence intervals. v1.0.1 corrects the documentation: the harness sets no decoding options, so models ran at Ollama default sampling (v1.0.0 README wrongly said temperature 0). Data and results are unchanged. Related work (added in v1.1.0). Cihan, İçöz, Haratian and Tüzün (arXiv 2505.20206, "Evaluating Large Language Models for Code Review") test GPT-4o and Gemini 2.0 Flash as code reviewers on correctness classification and report that performance depends on the type of code and that the outputs can be faulty. CryptoBench asks a related question for security review of cryptographic API use with local open models. It reports detection together with the matched secure controls, so a model that flags everything can be told apart from one that finds real misuse. v1.1.0 changes this description only. Data, harness and results are unchanged. v1.1.1 (2026-10-29) is a packaging release: this archive replaces the duplicate v1.0.1 file attached to v1.1.0 and updates the version metadata. Snippets, harness and results are byte-identical to v1.0.0. Cite as: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23067051 (all versions) Paper using this data: Chokshi, S. (2026). Known-Bad Names, Unknown-Bad Uses [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23113860 Rerun it: python3 crypto_bench.py --model <ollama-model> --repeats 5 (harness in the GitHub repository). Hugging Face: https://huggingface.co/datasets/sunny-chokshi/cryptobench