llama-7b-hf

This repo contains a low-rank adapter for LLaMA-7b fit on the Stanford Alpaca dataset.

This version of the weights was trained with the following hyperparameters:

Epochs: 10 (load from best epoch)
Batch size: 128
Cutoff length: 512
Learning rate: 3e-4
Lora r: 16
Lora target modules: q_proj, k_proj, v_proj, o_proj
That is:

python finetune.py \
--base_model='decapoda-research/llama-7b-hf' \
--num_epochs=10 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./lora-alpaca-512-qkvo' \
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
--lora_r=16 \
--micro_batch_size=8

搜集汇总

背景与挑战

背景概述

该数据集包含一个基于Stanford Alpaca数据集训练的LLaMA-7b低秩适配器，训练超参数包括epochs为10、batch size为128、cutoff length为512等。数据集由多个.bin文件和一个merge.json文件组成，总大小超过10GB。