Replication Package for 'Probing Coding Benchmark Memorization in Black-box LLMs with Vague Inputs'

Leakage Study — Raw Model Outputs Companion data for the paper "Probing Coding Benchmark Memorization in Black-box LLMs with Vague Inputs". This archive contains the raw model outputs used to detect benchmark memorization in code-generation LLMs. Scope. We probe seven flagship LLMs on five popular coding benchmarks under seven hint conditions, producing 245 JSONL files and 64,330 model responses in total. Benchmarks (data/raw/{benchmark}/): - humaneval (164 tasks), humanevalr (HumanEval+, perturbed, 164 tasks) - mbpp (sanitized, 257 tasks), mbppr (MBPP+, perturbed, 257 tasks) - livebench (coding subset, 50 tasks) Models ({benchmark}/{model}/): claude-opus-4-6, deepseek-v3-2-251201, gemini-3.1-pro-preview, glm-5.1, gpt-5.4, kimi-k2.5, and one anonymized model. Hint conditions ({model}/{hint}/data.jsonl): a single task-id-only template crossed with seven cumulative hints — H0 (no hint baseline), code_1–code_3 (skeleton, signature, test cases), and nl_1–nl_3 (keywords, description fragment, noisy description). Record schema. Each line in data.jsonl is one model response, containing metadata (task_id, exact prompt sent, entry point, canonical solution, test, template, hint), the model's raw result, and the model name. See README.md for the full schema. License. Released under Apache License 2.0.