遇见数据集

Final Sampled Test Sets for Benchmarking Local LLM Inference Platforms (Ollama, LM Studio, llama.cpp)

收藏
Zenodo2026-07-16 更新2026-08-01 收录
官方服务:

资源简介:

This dataset contains the final sampled test sets and prompt templates used in a controlled full-factorial benchmark comparing three local LLM inference runners (Ollama, LM Studio, and llama.cpp) across four quantization levels and three model architectures on Apple Silicon. Contents:- summarization_indosum_100.json — 100 documents sampled from the IndoSum fold-1 test set (text summarization task)- sentiment_smsa_200.json — 200 samples (stratified by class) from the IndoNLU SmSA test set (sentiment analysis: positive/negative/neutral)- classification_emot_200.json — 200 samples (stratified by class) from the IndoNLU EmoT test set (emotion classification: sadness/anger/love/fear/happy)- prompt_templates.json — the exact system and user prompt templates applied identically across all runners Sampling methodology: all samples were drawn with a fixed random seed (42); classification tasks used stratified sampling to preserve the original class distribution of the source datasets. Provenance: these are derived samples, not the original datasets. Source data:- IndoSum: Kurniawan, K. & Louvan, S. (2018). IndoSum: A New Benchmark Dataset for Indonesian Text Summarization. IALP 2018.- IndoNLU: Wilie, B. et al. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding. Full experimental code, raw benchmark results, and reproduction instructions: https://github.com/RoyanHabibie/benchmarking_local_llm The accompanying paper is in preparation; a citable reference will be added upon submission/publication.

提供机构:
Zenodo
创建时间:
2026-07-16
二维码
社区交流群
二维码
科研交流群
商业服务