samscrack/solidity-eval-2026

Solidity Eval (2026)是一个代理性Solidity基准测试数据集，专注于智能合约代码生成和评估。数据集包含任务，其中代理修改Etherscan验证的合约，将函数体替换为revert语句，然后构建和测试更改。奖励基于差异模糊测试通过率。数据集旨在与hermes-agent环境一起使用，但设计为与任何可以处理tarball的代理框架兼容。它包括两个配置，lite和full，具有不同的行数和选择标准。模式详细描述了数据集中的字段，如task_id、contract_name和各种与pragma相关的字段。README还解释了奖励语义、B2/B3奖励黑客缓解措施、来源以及从源语料库跳过的行。

Solidity Eval (2026) is an agentic Solidity benchmark focused on smart contract code generation and evaluation. The dataset involves tasks where agents modify Etherscan-verified contracts by replacing a function body with a revert statement, then build and test the changes. The reward is based on the differential-fuzz pass rate. The dataset is intended for use with the hermes-agent environment but is designed to be harness-agnostic. It includes two configs, lite and full, with different row counts and selection criteria. The schema details the fields in the dataset, such as task_id, contract_name, and various pragma-related fields. The README also explains the reward semantics, B2/B3 reward-hack mitigations, provenance, and skipped rows from the source corpus.