An Extension to BigCodeBench for Task Characterization and LLM-based Agent Evaluation

This upload contains two complementary datasets derived from "BigCodeBench". The first dataset focuses on task characterization, while the second contains information about LLM agents, their responses to the coding tasks, and the corresponding LLM-as-a-judge evaluations. Task characterization dataset The task characterization dataset extends BigCodeBench by categorizing Python coding tasks by domain, layer and complexity, to support research on task allocation. The original information has been preserved, except for the three columns: “test,” “entry_point,” and “doc_struct”; which were removed because they are not required for our work.This version only has information about the coding tasks. It can therefore be used by other researchers to develop approaches to task analysis and comparison, difficulty estimation, task categorization, and classification by skill or domain.The categorization was done by an LLM (model: OpenAI GPT-4o) using the following prompt:"You are a software engineering task classifier.Your job is to analyze a Python development task description and classify it according to several categories.Return the classification in JSON format.Categories:1. layerPossible values:- backend- frontend- data- infrastructure- integration- library 2. domainPossible values:- data_science- machine_learning- security- cryptography- natural_language_processing- optimization- networking- file_processing- image_processing- simulation- automation- web_scraping- database- api- visualization- none 3. complexityPossible values:- trivial- small- medium- large- research Instructions:- Choose the most appropriate value for each category.- Use none only when the code does not belong to any listed domain.- Return ONLY one minified JSON object with keys exactly:{{"layer":"...","domain":"...","complexity":"..."}} Task description:{task_description}Task libraries:{libs}" LLM agent evaluation dataset The LLM agent evaluation dataset contains information about the evaluated agents, their responses to each coding task, and the assessments produced by an LLM judge. It can be used to compare agent performance across tasks and task categories, analyze differences in response quality, and investigate the relationship between task characteristics and agent performance. The dataset was constructed by running four LLM-based coding agents on a selection of BigCodeBench tasks from three domains: data science, file processing, and machine learning. Three of the agents were configured as domain specialists and supported by domain-specific Retrieval-Augmented Generation (RAG). The other agent was a general-purpose baseline without RAG. All agents were based on the same underlying model (meta-llama/llama-3.1-8b-instruct), enabling the effects of agent specialization and domain-specific retrieval to be studied. For the RAG-supported agents, relevant documentation was collected, divided into overlapping text chunks, and indexed using dense vector embeddings. For each task, it was retrieved potentially relevant chunks from the corresponding domain-specific collection. An LLM-based filtering stage then assessed the relevance of the retrieved chunks. The model was "openai/gpt-4o-mini" and this was the prompt: "You are a chunk relevance evaluator for a Retrieval-Augmented Generation system. Your goal is to decide whether the retrieved chunk is useful for solving the coding task. Task domain: {domain} Coding task: {task} Retrieved chunk: {chunk_text} Evaluate the chunk according to: 1. Does it contain information useful for solving the task? 2. Is it aligned with the task domain? 3. Does it include concrete API usage, examples, function names, code patterns, or implementation details? 4. Is it specific enough to help generate correct Python code? 5. Should it be kept as context for the coding agent? Prefer chunks that contain concrete Python code examples, function signatures, method names, parameters, return types, or usage patterns. Penalize chunks that are only navigation text, API index pages, release notes, general explanations, or unrelated examples. Return only valid JSON with this format: {{ "relevance": 0.0, "useful": false, "reason": "short explanation" }} Scoring guide: - 1.0: directly useful and highly aligned with the task - 0.7: useful but only partially aligned - 0.4: weakly related or generic - 0.1: irrelevant, noisy, or misleading" When no retrieved content met the minimum relevance threshold, the agent generated its response without additional context. Each agent received the task instructions, starter code or function signature, and information about the required libraries. The specialized agents used two separate prompts: a retrieval query to identify relevant documentation and a generation prompt to produce the final task response. Retrieval query: "Coding task: {task} Task domain: {domain_label} Relevant libraries/imports: {libs} Retrieve documentation chunks with: - concrete Python API usage - examples - function names - implementation patterns - details useful for solving this exact task" Generation prompt for RAG agents: "You are a Python coding expert specialized in {domain_label}. Rules: - Solve the task according to its exact requirements. - Prefer appropriate techniques and libraries for {domain_label}. - If another library is necessary for correctness, use it. - Follow the starter imports and required function signature exactly. - Return exactly the objects requested by the task. - Use the retrieved context only when it is relevant. - Ignore retrieved information that conflicts with the task. - Do not mention the retrieved context in the answer. - Return code only, unless the task explicitly asks for explanation. Retrieved documentation context: {context}" The general-purpose no-RAG agent received the following prompt: "You are a general Python coding assistant. Rules: - Solve the task according to its exact requirements. - Follow the starter imports and required function signature exactly. - Return exactly the objects requested by the task. - Use appropriate Python libraries when helpful. - Do not use retrieved context. - Return code only, unless the task explicitly asks for explanation." The resulting code generation was then independently assessed by an LLM judge (model: openai/gpt-4o-mini) using seven criteria: correctness, completeness, complexity, maintainability, style, security, and efficiency. The evaluation includes individual scores for these criteria, their associated weights, an overall trustworthiness score, and recommendations for improving the generated answer. Here is the prompt for the LLM-as-a-judge evaluation: "You are an expert in software engineer trained to verify code correctness and assess its quality using software metrics such as cyclomatic complexity, comment-to-code ratio, coupling/cohesion, lines of code, as well as software security standards and practices (e.g., OWASP Top 10). Evaluate the following AI-generated code considering the criteria below. Give each criterion a score between 0.0 and 1.0, build a trustworthiness model by assigning weights to each criterion based on the relative criticality inferred from the code domain (e.g., security-critical system vs. UI script), and compute the overall trustworthiness score (0.0–1.0). Explain the weighting rationale in hidden reasoning (not in the output). If {task} lacks a clear specification, infer the intended functionality from the code and evaluate accordingly. When metric computation is not possible, qualitatively estimate values from the code structure and organization. Evaluation Criteria Criteria 1: Correctness • 1.0: Fully correct; meets the specification, handles edge cases, no major logical errors. • 0.7: Mostly correct but with minor logic/syntax or edge-case omissions. • 0.4: Partial solution; major logic errors or missing functionality. • 0.1: Incorrect or largely irrelevant solution. Criteria 2: Completeness • 1.0: All requirements addressed; includes proper error/edge-case handling. • 0.7: Most requirements addressed; some omissions or limited coverage. • 0.4: Only core functionality covered; major parts missing. • 0.1: Minimal or superficial attempt. Criteria 3: Complexity Evaluate using software metrics: Cyclomatic complexity, essential complexity, and size metrics. Estimate qualitatively if not measurable. • 1.0: Excellent — low complexity, simple control flow. • 0.7: Good — manageable branching or structure. • 0.4: Fair — noticeable complexity or large, hard-to-read code. • 0.1: Poor — deeply nested or hard to follow. Criteria 4: Maintainability Evaluate using coupling, cohesion, and comment-to-code ratio. • 1.0: Excellent — low coupling, high cohesion, well-documented. • 0.7: Good — minor maintainability issues. • 0.4: Fair — structural or documentation issues. • 0.1: Poor — high coupling, low cohesion, unclear structure. Criteria 5: Style • 1.0: Clean, idiomatic, consistent naming and formatting. • 0.7: Mostly clean with minor style issues. • 0.4: Several style or formatting problems. • 0.1: Poor readability or inconsistent style. Criteria 6: Security Evaluate code for i)Vulnerability coverage (injection, XSS, access control, misconfiguration, etc.), ii) Secure design (input validation, auth, secrets, crypto), ii) Runtime safety (config, resources, privileges, error handling) • 1.0: Secure — no known weaknesses. • 0.7: Minor missing mitigations. • 0.4: Some insecure practices. • 0.1: Vulnerable or unsafe. Criteria 7: Efficiency • 1.0: Optimized, scalable, minimal overhead. • 0.7: Reasonably efficient. • 0.4: Noticeable inefficiencies. • 0.1: Highly inefficient. Output Format Return only a JSON object, followed by a short recommendations section. Output format: {{ "correctness": 0.0, "completeness": 0.0, "complexity": 0.0, "maintainability": 0.0, "style": 0.0, "security": 0.0, "efficiency": 0.0, "weights": {{ "correctness": 0.0, "completeness": 0.0, "complexity": 0.0, "maintainability": 0.0, "style": 0.0, "security": 0.0, "efficiency": 0.0 }}, "overall_trustworthiness": 0.0 }} After the JSON, provide a “Recommendations” section summarizing key improvements for each criterion (maximum 3 lines per criterion). Output must be: • Valid JSON (no markdown, no explanations) • Plain text recommendations after JSON Answer to evaluate: {answer}"