ShahzebKhoso/local-code-arena-mbpp-starcoder_3b

该数据集名为Local Code Arena Telemetry: MBPP Benchmark on StarCoder 3B (Base)，是一个用于代码生成评估的遥测数据集。它包含了在传统StarCoder 3B基础模型上运行MBPP（Mostly Basic Python Problems）基准测试的原始评估指标、执行遥测日志和结构语法输出。数据集旨在记录自动基准测试环境中中层原始基础权重的行为动态，作为指令对齐影响分析的干净基线。具体内容包括：核心性能摘要（如功能Pass@1准确率为0.6%，平均生成速度为203.93令牌/秒）、参数与对齐矩阵对比（与指令对齐模型如qwen2.5-coder:3b的性能差异）、硬件配置（如使用NVIDIA GeForce RTX 4090 GPU）、数据集架构（包括任务ID、提示、参考代码、测试断言、模型元数据、原始生成、解析代码和评估指标等列字段）以及评估指标细分（如功能通过、沙箱反馈、代码BLEU分数、生成速度和延迟）。数据集可通过Hugging Face的datasets库加载，用于本地评估分析。This dataset, named Local Code Arena Telemetry: MBPP Benchmark on StarCoder 3B (Base), is a telemetry dataset for code generation evaluation. It hosts the raw evaluation metrics, execution telemetry logs, and structural syntax outputs captured from running the Mostly Basic Python Problems (MBPP) benchmark against the legacy StarCoder 3B base foundational model. The dataset documents the behavioral dynamics of mid-tier raw foundational weights inside automated benchmarking environments, serving as a clean baseline for instruction-alignment impact analysis. Key contents include: a core performance summary (e.g., functional Pass@1 accuracy of 0.6%, average generation speed of 203.93 tokens/second), a parameter and alignment matrix comparing performance with instruction-aligned models like qwen2.5-coder:3b, hardware configuration (e.g., using an NVIDIA GeForce RTX 4090 GPU), dataset architecture (with column fields such as task_id, prompt, canonical_reference, test_assertions, model_metadata, raw_generation, parsed_code, and evaluation_metrics), and a breakdown of evaluation metrics (e.g., functional_pass, sandbox_feedback, codebleu_overall, generation_speed_tps, latency_seconds). The dataset can be loaded via the Hugging Face datasets library for local evaluation analysis.