Fsoft-AIC/RepoExec

RepoExec是一个新颖的基准，旨在通过强调可执行性和正确性来评估仓库级别的代码生成。该基准通过关注现实世界的适用性，并提供代码功能的全面评估，填补了现有系统的空白。它旨在为代码功能和与开发者意图的一致性提供全面的评估，为现实场景中更可靠和适用的CodeLLMs铺平道路。

原始信息汇总

数据集概述

RepoExec 是一个新颖的基准测试，旨在以可执行性和正确性为重点，在仓库级别评估代码生成。该基准测试强调现实世界的适用性，并提供对代码功能的全面评估，旨在为更可靠和适用的 CodeLLMs 铺平道路。

支持的任务

RepoExec 支持仓库级别的代码生成，重点关注可执行性、测试用例的正确性以及跨文件依赖的上下文使用。

语言

目前，RepoExec 支持 Python 仓库。

数据集结构

数据实例

json { "id": 0, "project": "test-apps/python-string-utils", "module": "string_utils.manipulation", "entry_point": "reverse", "solution": "def reverse(input_string: str) -> str: """ Returns the string with its chars reversed.

*Example:*

>>> reverse(hello) # returns olleh

:param input_string: String to revert.
:type input_string: str
:return: Reversed string.
"""
if not is_string(input_string):
    raise InvalidInputError(input_string)

return input_string[::-1]",
"prompt": "import base64

import random import unicodedata import zlib from typing import Union from uuid import uuid4 from ._regex import * from .errors import InvalidInputError from .validation import is_snake_case, is_full_string, is_camel_case, is_integer, is_string

class InvalidInputError(TypeError): """ Custom error raised when received object is not a string as expected. """

def __init__(self, input_data: Any):
    """
    :param input_data: Any received object
    """
    type_name = type(input_data).__name__
    msg = Expected "str", received "{}".format(type_name)
    super().__init__(msg)

def is_string(obj: Any) -> bool: """ Checks if an object is a string.

*Example:*

>>> is_string(foo) # returns true
>>> is_string(bfoo) # returns false

:param obj: Object to test.
:return: True if string, false otherwise.
"""
return isinstance(obj, str)

def reverse(input_string: str) -> str: """ Returns the string with its chars reversed.

*Example:*

>>> reverse(hello) # returns olleh

:param input_string: String to revert.
:type input_string: str
:return: Reversed string.
"""

", "target_function_prompt": "def reverse(input_string: str) -> str: """ Returns the string with its chars reversed.

*Example:*

>>> reverse(hello) # returns olleh

:param input_string: String to revert.
:type input_string: str
:return: Reversed string.
"""

", "function_signature": "def reverse(input_string: str) -> str:", "docstring": " Returns the string with its chars reversed.

Example:

reverse(hello) # returns olleh

:param input_string: String to revert. :type input_string: str :return: Reversed string. ", "original_docstring": """" Returns the string with its chars reversed.

Example:

reverse(hello) # returns olleh

:param input_string: String to revert. :type input_string: str :return: Reversed string. """", "docstring_tokens": [ "Returns", "the", "string", "with", "its", "chars", "reversed", ".", "", "Example", ":", "", ">>>", "reverse", "(", "", "hello", "", ")", "#", "returns", "", "olleh", "", ":", "param", "input_string", ":", "String", "to", "revert", ".", ":", "type", "input_string", ":", "str", ":", "return", ":", "Reversed", "string", "." ], "cross_context": true, "isContained": false, "raw_solution": "def reverse(input_string: str) -> str: """ Returns the string with its chars reversed.

*Example:*

>>> reverse(hello) # returns olleh

:param input_string: String to revert.
:type input_string: str
:return: Reversed string.
"""
if not is_string(input_string):
    raise InvalidInputError(input_string)

return input_string[::-1]",
"check": "

import sys sys.path.insert(1, "/input/test-apps/python-string-utils") import unittest, pytest import math import random import re import copy import datetime import itertools import collections import heapq import statistics import functools import hashlib import numpy import numpy as np import string from typing import * from collections import * import pickle import timeout_decorator

all = [ camel_case_to_snake, snake_case_to_camel, reverse, shuffle, strip_html, prettify, asciify, slugify, booleanize, strip_margin, compress, decompress, roman_encode, roman_decode, ]

import base64 import random import unicodedata import zlib from typing import Union from uuid import uuid4

from string_utils._regex import * from string_utils.errors import InvalidInputError from string_utils.validation import is_snake_case, is_full_string, is_camel_case, is_integer, is_string

class __RomanNumbers: # internal rule mappings for encode() __mappings = [ # units {1: I, 5: V}, # tens {1: X, 5: L}, # hundreds {1: C, 5: D}, # thousands {1: M}, ]

# swap key/value definitions for decode()
__reversed_mappings = [{v: k for k, v in m.items()} for m in __mappings]

@classmethod
def __encode_digit(cls, index: int, value: int) -> str:
    # if digit is zero, there is no sign to display
    if value == 0:
        return 

    # from 1 to 3 we have just to repeat the sign N times (eg: III, XXX...)
    if value <= 3:
        return cls.__mappings[index][1] * value

    # if 4 we have to add unit prefix
    if value == 4:
        return cls.__mappings[index][1] + cls.__mappings[index][5]

    # if is 5, is a straight map
    if value == 5:
        return cls.__mappings[index][5]

    # if 6, 7 or 8 we have to append unit suffixes
    if value <= 8:
        suffix = cls.__mappings[index][1] * (value - 5)
        return cls.__mappings[index][5] + suffix

    # if 9 we have to prepend current unit to next
    return cls.__mappings[index][1] + cls.__mappings[index + 1][1]

@classmethod
def encode(cls, input_number: Union[str, int]) -> str:
    # force input conversion to a string (we need it in order to iterate on each digit)
    input_string = str(input_number)

    if not is_integer(input_string):
        raise ValueError(Invalid input, only strings or integers are allowed)

    value = int(input_string)

    if value < 1 or value > 3999:
        raise ValueError(Input must be >= 1 and <= 3999)

    input_len = len(input_string)
    output = 

    # decode digits from right to left (start from units to thousands)
    for index in range(input_len):
        # get actual digit value as int
        digit = int(input_string[input_len - index - 1])

        # encode digit to roman string
        encoded_digit = cls.__encode_digit(index, digit)

        # prepend encoded value to the current output in order to have the final string sorted
        # from thousands to units
        output = encoded_digit + output

    return output

@classmethod
def __index_for_sign(cls, sign: str) -> int:
    for index, mapping in enumerate(cls.__reversed_mappings):
        if sign in mapping:
            return index

    raise ValueError(Invalid token found: "{}".format(sign))

@classmethod
def decode(cls, input_string: str) -> int:
    if not is_full_string(input_string):
        raise ValueError(Input must be a non empty string)

    # reverse the provided string so that we can start parsing from units to thousands
    reversed_string = reverse(input_string.upper())

    # track last used value
    last_value = None

    # computed number to return
    output = 0

    # for each sign in the string we get its numeric value and add or subtract it to the computed output
    for sign in reversed_string:
        # are we dealing with units, tens, hundreds or thousands?
        index = cls.__index_for_sign(sign)

        # its basically 1 or 5 (based on mapping rules definitions)
        key_value = cls.__reversed_mappings[index][sign]

        # Based on the level (tens, hundreds...) we have to add as many zeroes as the level into which we are
        # in order to have the actual sign value.
        # For instance, if we are at level 2 we are dealing with hundreds, therefore instead of 1 or 5, we will
        # obtain 100 or 500 by adding 2 zeroes
        sign_value = int(str(key_value) + 0 * index)

        # increase total value if we are moving on with level
        if last_value is None or sign_value >= last_value:
            output += sign_value

        # Decrease value if we are back to a previous level
        # For instance, if we are parsing "IX", we first encounter "X" which is ten then "I" which is unit,
        # So we have to do the following operation in order to get 9 (the final result): 10 - 1
        else:
            output -= sign_value

        last_value = sign_value

    return output

class __StringCompressor:

@staticmethod
def __require_valid_input_and_encoding(input_string: str, encoding: str):
    if not is_string(input_string):
        raise InvalidInputError(input_string)

    if len(input_string) == 0:
        raise ValueError(Input string cannot be empty)

    if not is_string(encoding):
        raise ValueError(Invalid encoding)

@classmethod
def compress(cls, input_string: str, encoding: str = utf-8, compression_level: int = 9) -> str:
    cls.__require_valid_input_and_encoding(input_string, encoding)

    if not isinstance(compression_level, int) or compression_level < 0 or compression_level > 9:
        raise ValueError(Invalid compression_level: it must be an "int" between 0 and 9)

    # turns input string into a sequence of bytes using provided encoding
    original_bytes = input_string.encode(encoding)

    # compress bytes using zlib library
    compressed_bytes = zlib.compress(original_bytes, compression_level)

    # encode compressed bytes using base64
    # (this ensure that all characters will be available and that the output string can be used safely in any
    # context such URLs)
    encoded_bytes = base64.urlsafe_b64encode(compressed_bytes)

    # finally turns base64 bytes into a string
    output = encoded_bytes.decode(encoding)

    return output

@classmethod
def decompress(cls, input_string: str, encoding: str = utf-8) -> str:
    cls.__require_valid_input_and_encoding(input_string, encoding)

    # turns input string into a sequence of bytes
    # (the string is assumed to be a previously compressed string, therefore we have to decode it using base64)
    input_bytes = base64.urlsafe_b64decode(input_string)

    # decompress bytes using zlib
    decompressed_bytes = zlib.decompress(input_bytes)

    # decode the decompressed bytes to get the original string back
    original_string = decompressed_bytes.decode(encoding)

    return original_string

class __StringFormatter: def init(self, input_string): if not is_string(input_string): raise InvalidInputError(input_string)

    self.input_string = input_string

def __uppercase_first_char(self, regex_match):
    return regex_match.group(0).upper()

def __remove_duplicates(self, regex_match):
    return regex_match.group(1)[0]

def __uppercase_first_letter_after_sign(self, regex_match):
    match = regex_match.group(1)
    return match[:-1] + match[2].upper()

def __ensure_right_space_only(self, regex_match):
    return regex_match.group(1).strip() +  

def __ensure_left_space_only(self, regex_match):
    return   + regex_match.group(1).strip()

def __ensure_spaces_around(self, regex_match):
    return   + regex_match.group(1).strip() +  

def __remove_internal_spaces(self, regex_match):
    return regex_match.group(1).strip()

def __fix_saxon_genitive(self, regex_match):
    return regex_match.group(1).replace( , ) +  

# generates a placeholder to inject temporary into the string, it will be replaced with the original
# value at the end of the process
@staticmethod
def __placeholder_key():
    return $ + uuid4().hex + $

def format(self) -> str:
    # map of temporary placeholders
    placeholders = {}
    out = self.input_string

    # looks for url or email and updates placeholders map with found values
    placeholders.update({self.__placeholder_key(): m[0] for m in URLS_RE.findall(out)})
    placeholders.update({self.__placeholder_key(): m for m in EMAILS_RE.findall(out)})

    # replace original value with the placeholder key
    for p in

搜集汇总

数据集介绍

构建方式

RepoExec数据集由Fsoft-AI4Code团队构建，旨在填补现有代码生成基准在仓库级别可执行性评估方面的空白。其构建过程基于对真实世界Python仓库的深度剖析，从多个开源项目中精心选取了355个具有代表性的函数作为评估样本。每个样本均包含完整的上下文信息，如项目结构、模块依赖、函数签名、文档字符串及跨文件依赖关系，并配备了可执行的测试用例以验证生成代码的正确性。数据集按上下文粒度划分为full_context、medium_context和small_context三个子集，以模拟不同信息量下的代码生成场景。

特点

RepoExec数据集的核心特点在于其对代码可执行性与功能正确性的严格考量，超越了传统的简单匹配或静态评估。它强调仓库级别的代码生成，要求模型不仅理解局部函数逻辑，还需掌握跨文件的依赖关系，从而更贴近实际开发场景。每个数据点都提供了丰富的元数据，包括覆盖率指标、测试列表以及原始解决方案，使得评估结果具有高度的可重复性和细粒度的可解释性。此外，数据集支持多种上下文级别的配置，为研究不同信息量对代码生成质量的影响提供了便利。

使用方法

使用RepoExec数据集时，研究者可依据需求选择合适的上下文子集（full、medium或small）进行模型评估。数据以HuggingFace Datasets格式存储，可通过标准的load_dataset接口加载。每个样本包含prompt字段，可直接作为输入提供给代码生成模型，生成的代码需与对应的test_list中的测试用例进行比对，以验证其可执行性及功能正确性。此外，数据集还提供了check字段，内含完整的测试执行环境配置，便于自动化流水线集成。详细的评估脚本和指南可在其GitHub仓库中找到。

背景与挑战

背景概述

随着大语言模型在代码生成任务中的广泛应用，现有的基准测试多聚焦于函数级或片段级的代码合成，难以真实反映模型在复杂仓库环境下的表现。为此，FPT软件人工智能中心的研究团队于2024年提出了RepoExec数据集，旨在构建一个仓库级别的可执行代码生成评估基准。该数据集以Python仓库为对象，通过引入跨文件依赖上下文，评估模型在真实软件工程场景中生成可运行、语义正确代码的能力。RepoExec的出现填补了现有基准在可执行性和仓库级一致性评估方面的空白，为代码大模型的实际部署与可靠性验证提供了重要支撑，对推动代码智能研究向工业应用迈进具有深远意义。

当前挑战

RepoExec所面临的挑战主要体现在两个层面。首先，在领域问题层面，现有代码生成评估多停留在静态匹配或单元测试层面，缺乏对仓库级跨文件依赖和运行时正确性的考量，导致模型在真实项目中的表现被高估。RepoExec通过设计包含完整导入链与执行环境的测试用例，迫使模型生成的代码必须满足项目级约束，这对模型理解长程依赖和项目结构的能力提出了严苛要求。其次，在构建过程中，如何从开源仓库中精确提取函数签名、文档字符串及对应的跨文件上下文，并构造出既具代表性又不会泄露目标函数的测试用例，是一项艰巨任务。此外，确保每个样本的可执行性同时避免环境差异导致的假阴性，需要精细化的沙盒设计与依赖管理，这显著增加了数据集的构建复杂度与维护成本。

常用场景

经典使用场景

RepoExec 作为面向仓库级别的代码生成评估基准，其核心应用场景在于全面衡量代码大模型在真实软件工程环境中的函数级代码生成能力。该数据集通过精心设计的多粒度上下文分割（full_context、medium_context、small_context），模拟了开发者在不同信息可用性条件下编写代码的典型情境。研究者可利用该基准，精准评估模型在依赖跨文件上下文时的代码可执行性、测试用例通过率以及功能正确性，从而深入洞悉模型对仓库级代码结构和依赖关系的理解程度。

解决学术问题

RepoExec 致力于填补现有代码生成基准在真实应用场景中的关键空白。传统的评估方法往往局限于单函数或孤立片段的生成，忽略了实际项目中错综复杂的跨文件依赖与上下文关联，导致评估结果与模型在真实开发中的表现相去甚远。该数据集通过引入可执行性验证和细粒度的测试覆盖分析，解决了如何客观、可靠地衡量代码功能正确性与开发者意图对齐度的学术难题，为构建更稳健、更实用的代码语言模型奠定了坚实的评估基础。

衍生相关工作

RepoExec 的诞生催生了一系列围绕仓库级代码理解与生成的前沿研究工作。其强调可执行性和跨文件上下文的设计理念，启发后续学者提出了针对代码大模型的上下文感知训练策略和基于执行反馈的强化学习方法。此外，该基准所公开的细粒度上下文分割和测试用例体系，为研究代码表示学习、依赖图建模以及程序修复等任务提供了标准化的评测平台，推动了整个领域从简单的语法匹配向深层的语义和执行正确性评估迈进。