ShahzebKhoso/local-code-arena-eval_results_deepseek-r1_8b

--- license: mit task_categories: - text-generation tags: - code - code-generation - evaluation - llm-evaluation - local-llm - mbpp - deepseek - deepseek-r1 - reasoning - telemetry size_categories: - n < 1K --- # Local Code Arena Telemetry: MBPP Benchmark on DeepSeek R1 8B This repository hosts the raw evaluation metrics, execution telemetry logs, and structural syntax outputs captured from running the **Mostly Basic Python Problems (MBPP)** benchmark against the **DeepSeek R1 8B** distilled reasoning architecture. This specific run establishes the operational baseline and performance characteristics of mid-tier reasoning models under strict automated evaluation and sandbox time limits on consumer hardware. ## 📊 Core Performance Summary - **Evaluation Target:** `deepseek-r1:8b` (via Ollama Server) - **Functional Pass@1 Accuracy:** **24.0%** - **Average Generation Speed:** **85.06 Tokens/Second** ⚡ - **Evaluation Window:** 500 tasks (Test Split) --- ## 📈 Reasoning vs. Specialization Matrix (~8B Scale) Placing this reasoning dataset alongside its size-matched counterparts highlights a stark contrast between dense logical reasoning paths and domain-specific code saturation: | Model Tag | Parameter Scale | Focus Class | Pass@1 Accuracy | Local Throughput (TPS) | | :--- | :--- | :--- | :--- | :--- | | `qwen2.5-coder:7b` | 7.2 Billion | Code Specialist | **51.0%** 🏆 | **68.33 Tokens/Sec** | | `qwen3:8b` | 8.2 Billion | Generalist Base | **24.6%** | **90.89 Tokens/Sec** | | **`deepseek-r1:8b`** | **8.0 Billion** | **Distilled Reasoning** | **24.0%** 🎯 | **85.06 Tokens/Sec** | *Key Technical Insight:* The telemetry shows that scaling within the distilled reasoner family from 1.5B to 8B drives a significant precision increase (from 13.4% to 24.0%). However, on basic functional tasks, the model's internal chain-of-thought overhead (`<think>`) acts as a bottleneck compared to specialized models. Qwen 2.5 Coder 7B maintains an absolute advantage by outputting code directly without extra processing tokens, avoiding sandbox timeouts and maximizing zero-shot functional execution. --- ## 💻 Baseline Hardware Configuration All telemetry records inside this dataset matrix were compiled on a singular local environment footprint: - **Host System:** Alienware m18 Performance Notebook - **GPU Accelerator:** NVIDIA GeForce RTX 4090 Laptop GPU (16GB GDDR6 VRAM / 175W TGP Max) - **Driver / CUDA Stack:** NVIDIA Driver 581.95 | CUDA 13.0 - **Isolation Engine:** Multi-threaded Python Code Execution Sandbox (2.0s Hard Wall-Clock Timeout Limit) --- ## 📂 Dataset Architecture & Feature Schema Each row within this dataset represents a fully evaluated, structured code generation instance. The table outlines the schemas available in the parquet records: | Column Field | Data Type | Functional Description | | :--- | :--- | :--- | | `task_id` | `int64` | The original source tracking pointer for the MBPP dataset entry. | | `prompt` | `string` | The text string instruction passed to the local LLM model instance. | | `canonical_reference` | `string` | The ground-truth standard Python solution provided by the base dataset. | | `test_assertions` | `list` | String arrays of explicit runtime python `assert` verification operations. | | `model_metadata` | `struct` | JSON dictionary tracking `model_id` and the hosting hardware parameters. | | `raw_generation` | `string` | The unedited, raw string return received directly from the local API stream. | | `parsed_code` | `string` | Extracted code block stripped cleanly of conversational markdown text wrappers. | | `evaluation_metrics` | `struct` | Deep metrics tracking structural and execution telemetry. | ### 🛠️ Evaluation Metrics Breakdown Inside the `evaluation_metrics` structural child frame, fields map precise tracking criteria: * **`functional_pass`** (`bool`): Evaluates to `true` if the code compiled cleanly and completed **100%** of the associated test assertion strings. * **`sandbox_feedback`** (`string`): The precise stdout message or traceback captured by the isolated runtime environment loop (e.g., `Execution Timeout`, `NameError`, or `Success`). * **`codebleu_overall`** (`float`): An aggregated structural score grading AST matches and data-flow syntax layout configurations against the ground truth target. * **`generation_speed_tps`** (`float`): The dedicated processing efficiency score capturing exact **Tokens per Second** generated on the local RTX 4090. * **`latency_seconds`** (`float`): The absolute round-trip execution latency for model inference response strings. --- ## 🚀 How to Utilize This Dataset You can stream this telemetry dataset into your local evaluation analysis notebooks using the Hugging Face `datasets` engine: ```python from datasets import load_dataset # Stream the local code arena performance log straight into your dataframe dataset = load_dataset("ShahzebKhoso/local-code-arena-mbpp-deepseek-r1-8b") # Access individual record blocks first_entry = dataset['train'][0] print(f"Recorded Matrix Throughput: {first_entry['evaluation_metrics']['generation_speed_tps']} TPS") ``` ## 📄 Licensing & Citation This dataset is distributed under the permissive MIT License. If you leverage these raw telemetry files in comparative research workflows, please point back to this Hub repository space.This repository hosts the raw evaluation metrics, execution telemetry logs, and structural syntax outputs captured from running the Mostly Basic Python Problems (MBPP) benchmark against the DeepSeek R1 8B distilled reasoning architecture. This specific run establishes the operational baseline and performance characteristics of mid-tier reasoning models under strict automated evaluation and sandbox time limits on consumer hardware. The dataset includes core performance summary (e.g., functional pass rate 24.0%, average generation speed 85.06 tokens/second), reasoning vs. specialization matrix comparison (with models like Qwen 2.5 Coder 7B), baseline hardware configuration (using NVIDIA RTX 4090 laptop GPU), dataset architecture and feature schema (including fields such as task_id, prompt, canonical_reference, test_assertions, model_metadata, raw_generation, parsed_code, and evaluation_metrics), and evaluation metrics breakdown (e.g., functional_pass, sandbox_feedback, codebleu_overall, generation_speed_tps, latency_seconds). The dataset can be loaded via the Hugging Face datasets library and is suitable for code generation evaluation, LLM performance analysis, and reasoning model research.

搜集汇总

数据集介绍

构建方式

本数据集是基于DeepSeek R1 8B蒸馏推理模型在MBPP基准测试上的完整评估记录。数据集构建过程中，通过Ollama服务器在本地消费级硬件（配备NVIDIA GeForce RTX 4090笔记本电脑GPU）上运行500个测试任务，并利用多线程Python代码执行沙箱（2.0秒硬时限）对模型生成的代码进行功能性验证。每一行数据记录了一个独立的代码生成实例，包含原始指令、标准参考解、测试断言、模型元数据、原始生成文本、解析后的代码块以及详细的评估指标，其中评估指标涵盖了功能通过率、沙箱反馈、CodeBLEU结构评分、生成速度与延迟等关键信息。

特点

该数据集的独特之处在于它提供了细粒度的执行遥测日志，能够精确反映蒸馏推理模型在本地环境中的真实表现。其核心特点包括：功能通过率仅为24.0%，平均生成速度达85.06 Tokens/秒；与同规模模型对比，显示出蒸馏推理模型内部链式思维开销对基础编程任务性能的制约。数据集的结构化特征架构包含task_id、prompt、canonical_reference、test_assertions、model_metadata等字段，尤其evaluation_metrics字段内部嵌套了functional_pass、sandbox_feedback、codebleu_overall、generation_speed_tps和latency_seconds等深度追踪指标，为研究推理模型的效率与准确性权衡提供了宝贵资源。

使用方法

用户可通过Hugging Face的datasets库直接加载该数据集，用于本地评估分析工作流。具体使用方法为：调用load_dataset("ShahzebKhoso/local-code-arena-mbpp-deepseek-r1-8b")函数即可将数据流式加载至数据框，然后可通过索引访问单个记录块，例如获取第一个条目的生成速度指标。该数据集适合用于对比不同模型在代码生成任务上的性能，分析推理开销对功能性正确率的影响，或作为本地LLM评估管道的基准数据源。

背景与挑战

背景概述

该数据集由研究者Shahzeb Khoso于近期构建，旨在系统评估中等规模推理模型在本地硬件上的代码生成性能。聚焦于DeepSeek R1 8B蒸馏推理架构在Mostly Basic Python Problems（MBPP）基准上的表现，该数据集通过严格的自动化评估与沙箱时限模拟，填补了消费者硬件环境下推理型代码模型基准测试的空白。其研究核心在于揭示链式思维推理在功能性代码任务中的效率权衡，并与同等参数规模的通用基座模型及代码专精模型进行对比。作为首批公开的本地推理模型代码生成遥测数据集，它为社区提供了可复现、细粒度的性能基线，对边缘计算与隐私敏感场景下的模型选型具有重要参考价值。

当前挑战

该数据集所解决的领域挑战在于：推理型模型在处理基础功能性代码任务时，其内部链式思维开销成为瓶颈，导致准确率显著低于专精代码模型——DeepSeek R1 8B的Pass@1准确率仅为24.0%，而同等规模的Qwen 2.5 Coder 7B高达51.0%。构建过程中面临的主要挑战包括：在RTX 4090笔记本GPU（16GB显存）的有限资源下，必须设计2.0秒硬墙钟超时的沙箱环境以平衡评估效率与安全性；同时需精确捕获并解析模型的原始生成输出，去除对话标记后提取可执行代码片段，并对每项任务记录完整的执行遥测指标（如生成速度、延迟、CodeBLEU分数），确保数据完整性与可复现性。

常用场景

经典使用场景

该数据集主要用于评估和比较中小规模推理模型在代码生成任务上的性能表现。具体而言，它记录了DeepSeek R1 8B模型在MBPP基准测试上的完整推理轨迹与执行日志，包括功能正确性、生成速度、CodeBLEU结构相似度等细粒度指标。研究者可利用这些标准化的评估结果，系统性地分析蒸馏推理模型在消费级硬件环境下的代码生成能力，为后续模型优化与算法改进提供可复现的基线参考。

解决学术问题

该数据集解决了学术界在小型推理模型评估中缺乏统一、细粒度基准的问题。通过提供涵盖功能正确性、生成效率与结构相似度的多维评估指标，它帮助研究者揭示蒸馏推理模型在代码生成任务中的性能瓶颈——如内部思维链开销导致的超时问题，以及模型规模与任务复杂性之间的非线性关系。这为理解推理架构在资源受限场景下的行为特征提供了关键洞察，推动更高效的轻量级模型设计。

衍生相关工作

该数据集衍生出的经典工作包括：其一，基于其记录的多维度评估指标，研究者可构建推理模型的性能预测框架，探索生成速度、模型规模与功能正确性之间的权衡曲线；其二，利用其细粒度telemetry日志，可设计针对代码生成任务的注意力机制优化策略，减少无效思维链开销；其三，该数据集常作为基线，对比验证新型蒸馏算法或结构化代码解码方法在相同硬件条件下的改进效果。