FVEval

该数据集名为FVEval，是首个全面评估框架，旨在表征大型语言模型（LLM）在数字硬件设计中的形式验证（FV）相关任务上的性能。该基准测试包含了生成SystemVerilog断言和对设计寄存器传输级（RTL）进行推理的任务，并提供全面和部分功能等价的度量标准。该数据集涵盖了广泛现有的LLM，包括专有和开源模型。其任务之一是评估语言模型根据自然语言描述生成SystemVerilog断言的能力。This dataset, named FVEval, is the first comprehensive evaluation framework aimed at characterizing the performance of Large Language Models (LLMs) on tasks related to formal verification (FV) in digital hardware design. This benchmark encompasses tasks of generating SystemVerilog assertions and conducting reasoning on design Register-Transfer Level (RTL) circuits, and provides both comprehensive and partial functional equivalence metrics. This dataset covers a wide range of existing LLMs, including both proprietary and open-source models. One of its tasks is to evaluate the capability of language models to generate SystemVerilog assertions based on natural language descriptions.

搜集汇总

背景与挑战

背景概述

FVEval是一个用于评估大型语言模型在硬件形式验证领域性能的数据集，由NVlabs发布。它包含三个子基准任务：NL2SVA-Human和NL2SVA-Machine专注于从自然语言指令生成SystemVerilog断言，而Design2SVA则涉及SystemVerilog测试台生成。数据集依赖商业工具Cadence Jasper进行完整评估，并提供了完整的代码和脚本以支持模型测试和基准生成。