neuralmagic/quantized-llama-3.1-humaneval-evals

--- configs: - config_name: Meta-Llama-3.1-8B-Instruct-W4A16 data_files: "Meta-Llama-3.1-8B-Instruct-W4A16.csv" - config_name: Meta-Llama-3.1-70B-Instruct-W4A16 data_files: "Meta-Llama-3.1-70B-Instruct-W4A16.csv" - config_name: Meta-Llama-3.1-405B-Instruct-W4A16 data_files: "Meta-Llama-3.1-405B-Instruct-W4A16.csv" - config_name: Meta-Llama-3.1-8B-Instruct-W8A8-INT8 data_files: "Meta-Llama-3.1-8B-Instruct-W8A8-INT8.csv" - config_name: Meta-Llama-3.1-70B-Instruct-W8A8-INT8 data_files: "Meta-Llama-3.1-70B-Instruct-W8A8-INT8.csv" - config_name: Meta-Llama-3.1-405B-Instruct-W8A8-INT8 data_files: "Meta-Llama-3.1-405B-Instruct-W8A8-INT8.csv" - config_name: Meta-Llama-3.1-8B-Instruct-W8A8-FP8 data_files: "Meta-Llama-3.1-8B-Instruct-W8A8-FP8.csv" - config_name: Meta-Llama-3.1-70B-Instruct-W8A8-FP8 data_files: "Meta-Llama-3.1-70B-Instruct-W8A8-FP8.csv" - config_name: Meta-Llama-3.1-405B-Instruct-W8A8-FP8 data_files: "Meta-Llama-3.1-405B-Instruct-W8A8-FP8.csv" --- # Coding Benchmark Results The coding benchmark results were obtained with the [EvalPlus](https://github.com/neuralmagic/evalplus) library. | | HumanEval<br>pass@1 | HumanEval+<br>pass@1 | | -------------------------------------------------- | :-----------------: | :------------------: | | meta-llama_Meta-Llama-3.1-405B-Instruct | 67.3 | 67.5 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic) | 66.7 | 66.6 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16) | 66.5 | 66.4 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a8) | 64.3 | 64.8 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic) | 58.1 | 57.7 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16) | 57.1 | 56.8 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8) | 56.0 | 56.6 | | meta-llama_Meta-Llama-3.1-70B-Instruct | 55.8 | 58.2 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8) | 27.6 | 26.7 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic) | 27.4 | 27.0 | | meta-llama_Meta-Llama-3.1-8B-Instruct | 25.1 | 26.5 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16) | 23.4 | 24.6 |配置项： - 配置名称：Meta-Llama-3.1-8B-Instruct-W4A16，数据文件："Meta-Llama-3.1-8B-Instruct-W4A16.csv" - 配置名称：Meta-Llama-3.1-70B-Instruct-W4A16，数据文件："Meta-Llama-3.1-70B-Instruct-W4A16.csv" - 配置名称：Meta-Llama-3.1-405B-Instruct-W4A16，数据文件："Meta-Llama-3.1-405B-Instruct-W4A16.csv" - 配置名称：Meta-Llama-3.1-8B-Instruct-W8A8-INT8，数据文件："Meta-Llama-3.1-8B-Instruct-W8A8-INT8.csv" - 配置名称：Meta-Llama-3.1-70B-Instruct-W8A8-INT8，数据文件："Meta-Llama-3.1-70B-Instruct-W8A8-INT8.csv" - 配置名称：Meta-Llama-3.1-405B-Instruct-W8A8-INT8，数据文件："Meta-Llama-3.1-405B-Instruct-W8A8-INT8.csv" - 配置名称：Meta-Llama-3.1-8B-Instruct-W8A8-FP8，数据文件："Meta-Llama-3.1-8B-Instruct-W8A8-FP8.csv" - 配置名称：Meta-Llama-3.1-70B-Instruct-W8A8-FP8，数据文件："Meta-Llama-3.1-70B-Instruct-W8A8-FP8.csv" - 配置名称：Meta-Llama-3.1-405B-Instruct-W8A8-FP8，数据文件："Meta-Llama-3.1-405B-Instruct-W8A8-FP8.csv" # 编码基准测试结果 本次编码基准测试结果通过EvalPlus（https://github.com/neuralmagic/evalplus）库生成。 | | HumanEval<br>pass@1 | HumanEval+<br>pass@1 | | -------------------------------------------------- | :-----------------: | :------------------: | | meta-llama_Meta-Llama-3.1-405B-Instruct | 67.3 | 67.5 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic) | 66.7 | 66.6 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16) | 66.5 | 66.4 | | [neuralmagic_Meta-Llama-3.1-405B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a8) | 64.3 | 64.8 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic) | 58.1 | 57.7 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16) | 57.1 | 56.8 | | [neuralmagic_Meta-Llama-3.1-70B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8) | 56.0 | 56.6 | | meta-llama_Meta-Llama-3.1-70B-Instruct | 55.8 | 58.2 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W8A8-INT8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8) | 27.6 | 26.7 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W8A8-FP8](https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic) | 27.4 | 27.0 | | meta-llama_Meta-Llama-3.1-8B-Instruct | 25.1 | 26.5 | | [neuralmagic_Meta-Llama-3.1-8B-Instruct-W4A16](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16) | 23.4 | 24.6 |