yujiepan/awq-model-zoo

---
tags:
- awq
- llm
- quantization
---

# yujiepan/awq-model-zoo

Here are some pre-computed awq information (scales & clips) used in [llm-awq](https://github.com/mit-han-lab/llm-awq).

## Scripts

- Install the forked `llm-awq` at [https://github.com/yujiepan-work/llm-awq/tree/a41a08e79d8eb3d6335485b3625410af22a74426](https://github.com/yujiepan-work/llm-awq/tree/a41a08e79d8eb3d6335485b3625410af22a74426). Note: works with transformers==4.35.2

- Generating awq-info.pt:

```bash
python do_awq.py --model_id mistralai/Mistral-7B-v0.1 --w_bit 8 --q_group_size 128 --dump_awq ./awq-info.pt
```

- Load a quantized model: You can use the offical repo to get a fake/real quantized model. Alternatively, you can load a fake-quantized model:

```python
from do_awq import FakeAWQModel
FakeAWQModel.from_pretrained('mistralai/Mistral-7B-v0.1', awq_meta_path='./awq-info.pt', output_folder='./tmp/')
```

Note: the code is not in good shape.

## Related links

- <https://huggingface.co/datasets/mit-han-lab/awq-model-zoo>

原始信息汇总

yujiepan/awq-model-zoo

数据集描述

该数据集包含一些预计算的awq信息（scales & clips），用于llm-awq项目。

使用方法

1. 安装forked的llm-awq：

   • 安装地址：https://github.com/yujiepan-work/llm-awq/tree/a41a08e79d8eb3d6335485b3625410af22a74426
   • 注意：适用于transformers==4.35.2

2. 生成awq-info.pt： bash python do_awq.py --model_id mistralai/Mistral-7B-v0.1 --w_bit 8 --q_group_size 128 --dump_awq ./awq-info.pt

3. 加载量化模型：

   • 使用官方仓库获取假/真实量化模型。
   • 或者加载假量化模型： python from do_awq import FakeAWQModel FakeAWQModel.from_pretrained(mistralai/Mistral-7B-v0.1, awq_meta_path=./awq-info.pt, output_folder=./tmp/)

注意事项

• 代码状态：代码质量不佳。