ledale/code_generation_lite

--- license: cc tags: - code - code generation pretty_name: LiveCodeBench size_categories: - n<1K --- ## LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code <p align="center"> <a href="https://livecodebench.github.io/">🏠 Home Page</a> • <a href="https://github.com/LiveCodeBench/LiveCodeBench">💻 GitHub Repository </a> • <a href="https://livecodebench.github.io/leaderboard.html">🏆 Leaderboard</a> • <a href="https://arxiv.org/abs/2403.07974">📄 Paper </a> </p>  ## Change Log Since LiveCodeBench is a continuously updated benchmark, we provide different versions of the dataset. Particularly, we provide the following versions of the dataset: - `release_v1`: The initial release of the dataset with problems released between May 2023 and Mar 2024 containing 400 problems. - `release_v2`: The updated release of the dataset with problems released between May 2023 and May 2024 containing 511 problems. - `release_v3`: The updated release of the dataset with problems released between May 2023 and Jul 2024 containing 612 problems. - `release_v4`: The updated release of the dataset with problems released between May 2023 and Sep 2024 containing 713 problems. - `release_v5`: The updated release of the dataset with problems released between May 2023 and Jan 2025 containing 880 problems. You can use the `version_tag` argument to load the desired version of the dataset. Additionally, you can use version tags like `v1`, `v2`, `v1_v3`, `v4_v5` to get the problems released in a specific version. ## Dataset Description LiveCodeBench is a "live" updating benchmark for holistically evaluating code related capabilities of LLMs. Particularly, it evaluates LLMs across a range of capabilties including code generation, self-repair, test output prediction, and code execution. This is the code generation scenario of LiveCodeBench. It is also used for evaluating self-repair using test case feedback. LiveCodeBench problems are collected from competition programming websites with particular focus on maintaining problem quality, test case quality, and problem difficulty diversity. This scenario currently hosts over 500 problems from LeetCode, AtCoder, and Codeforces. Each problem instance consists of a problem description, input/output examples, and hidden test cases. Additionally, every problem is tagged with its difficulty level and release date, which allows measuring model performance across different time windows. The goal is to generate a correct and efficient solution for each problem instance. The initial code_generation dataset included a larger number of test cases which leads to a substantially large dataset size. This (lite) version has pruned and sampled tests while trying to ensure similar performances with the original dataset. Going forward, livecodebench will be using this lite version for code generation evaluations. ## Usage You can use the dataset by loading it from the Hugging Face datasets library. Additionally, the version tag "release_v1" is used to specify the (temporal) version of the dataset. "v1" corresponds to the initial release of the dataset and "release_v2" is the second version. ```python from datasets import load_dataset lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2") ```--- license: CC tags: - 代码 - 代码生成 pretty_name: LiveCodeBench size_categories: - 样本数少于1000 --- # LiveCodeBench：面向代码场景的大语言模型（Large Language Model）全维度无数据污染评估基准 <p align="center"> <a href="https://livecodebench.github.io/">🏠 主页</a> • <a href="https://github.com/LiveCodeBench/LiveCodeBench">💻 GitHub 仓库</a> • <a href="https://livecodebench.github.io/leaderboard.html">🏆 排行榜</a> • <a href="https://arxiv.org/abs/2403.07974">📄 论文</a> </p>  ## 更新日志 由于LiveCodeBench是持续更新的基准数据集，我们提供了多个不同版本的数据集，具体版本如下： - `release_v1`：数据集首次发布版本，收录2023年5月至2024年3月间发布的题目，共400道。 - `release_v2`：数据集首次更新版本，收录2023年5月至2024年5月间发布的题目，共511道。 - `release_v3`：数据集第二次更新版本，收录2023年5月至2024年7月间发布的题目，共612道。 - `release_v4`：数据集第三次更新版本，收录2023年5月至2024年9月间发布的题目，共713道。 - `release_v5`：数据集第四次更新版本，收录2023年5月至2025年1月间发布的题目，共880道。 你可以通过指定`version_tag`参数加载所需版本的数据集。此外，你还可以使用`v1`、`v2`、`v1_v3`、`v4_v5`这类版本标签，获取对应版本区间内发布的题目。 ## 数据集说明 LiveCodeBench是一个“动态更新”的基准数据集，用于全维度评估大语言模型（Large Language Model）的代码相关能力。具体而言，该基准从代码生成、自我修复、测试输出预测、代码执行等多个维度对大语言模型进行评估。本数据集为LiveCodeBench的代码生成赛道，同时也可用于基于测试用例反馈的自我修复能力评估。 LiveCodeBench的题目均采集自编程竞赛平台，其设计重点在于保证题目质量、测试用例质量以及题目难度的多样性。本赛道目前收录了来自LeetCode、AtCoder、Codeforces的500余道题目。每道题目均包含题目描述、输入输出样例以及隐藏测试用例。此外，每道题目均标注了难度等级与发布日期，支持按不同时间窗口评估模型性能。任务目标为每道题目生成正确且高效的代码解决方案。 初始的代码生成数据集包含大量测试用例，导致数据集体积过大。本（轻量版）数据集对测试用例进行了裁剪与采样，同时尽可能保证与原数据集的评估结果一致性。后续LiveCodeBench将采用此轻量版本开展代码生成能力评估。 ## 使用方法 你可以通过Hugging Face Datasets库加载本数据集。此外，可通过版本标签`release_v1`指定数据集的（时间维度）版本：`v1`对应数据集首次发布版本，`release_v2`对应第二版数据集。 python from datasets import load_dataset lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")