Final Sampled Test Sets for Benchmarking Local LLM Inference Platforms (Ollama, LM Studio, llama.cpp)
收藏资源简介:
This dataset contains the final sampled test sets and prompt templates used in a controlled full-factorial benchmark comparing three local LLM inference runners (Ollama, LM Studio, and llama.cpp) across four quantization levels and three model architectures on Apple Silicon. Contents:- summarization_indosum_100.json — 100 documents sampled from the IndoSum fold-1 test set (text summarization task)- sentiment_smsa_200.json — 200 samples (stratified by class) from the IndoNLU SmSA test set (sentiment analysis: positive/negative/neutral)- classification_emot_200.json — 200 samples (stratified by class) from the IndoNLU EmoT test set (emotion classification: sadness/anger/love/fear/happy)- prompt_templates.json — the exact system and user prompt templates applied identically across all runners Sampling methodology: all samples were drawn with a fixed random seed (42); classification tasks used stratified sampling to preserve the original class distribution of the source datasets. Provenance: these are derived samples, not the original datasets. Source data:- IndoSum: Kurniawan, K. & Louvan, S. (2018). IndoSum: A New Benchmark Dataset for Indonesian Text Summarization. IALP 2018.- IndoNLU: Wilie, B. et al. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding. Full experimental code, raw benchmark results, and reproduction instructions: https://github.com/RoyanHabibie/benchmarking_local_llm The accompanying paper is in preparation; a citable reference will be added upon submission/publication.



