A Statistically Grounded Evaluation Protocol for Quantized Small Language Models on Memory-Constrained Edge Platforms

This artifact accompanies the manuscript "A Statistically Grounded Evaluation Protocol for Quantized Small Language Models on Memory-Constrained Edge Platforms" submitted to The Journal of Supercomputing. We evaluate Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct at seven precision levels (FP16, Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0) on two Chinese benchmarks (a C-Eval subset and a CMMLU subset) on a Raspberry Pi 5 (8 GB) and a desktop CPU, plus a 20+20 Chinese safety audit on the 1.5B model. Accuracy comparisons use two-proportion z-tests, Newcombe-Wilson difference intervals, Wilson confidence intervals, and a pre-specified non-inferiority lower-bound check. Package contents:- results/: all measurement JSON, including item-level predictions for the 0.5B C-Eval subset and the safety audit, and per-level summaries with confidence intervals for all configurations.- scripts/: quantization, evaluation, statistics, table and figure generation scripts.- prompts/: the 40 Chinese safety-audit prompts (20 harmful + 20 benign).- figures/: the nine manuscript figures in PNG, plus submission-ready TIF/EPS.- README.md: full documentation, reproduction steps, and license notes. Data note: item-level rows are stored for the 0.5B C-Eval subset and for the safety audit; for the 1.5B C-Eval and both CMMLU runs the on-device runner stored per-level summaries with per-repeat counts rather than per-item predictions, so those configurations are reported at the aggregate level only. Sampling is without replacement with a fixed seed set shared across quantization levels. C-Eval is used under CC BY-NC-SA 4.0, CMMLU under Apache-2.0 (ModelScope v1), and the Qwen2.5 weights under Apache-2.0. Our measurement data and scripts are released under CC BY 4.0.