Vezora/Mini_Orca_Uncencored_Alpaca

---
license: apache-2.0
---
This is dataset is a modified version of "psmathur's" Mini orca dataset, formated in the alpaca format and uncencored.

For ALPACA LORA users:
Modules you can target with lora:"gate_proj", "down_proj", "up_proj", "q_proj", "v_proj", "k_proj", "o_proj"
Most lora models use:"q_proj", "v_proj", "k_proj", "o_proj"
Platypus which got terrific results: "gate_proj", "down_proj", "up_proj"

Research on targeting certain modules still needs to be done, but if you don't want to train over a previously trained models newly learned abilities, target different modules than the ones used for original training.

Hyper perameters used by Platypus:
Hyperparameters for 13B and 70B Models
Hyperparameter Platypus2-13B / 70B
batch size 16
micro batch size 1
num epochs 1
learning rate 4e-4 / 3e-4
cutoff len 4096
lora rank 16
lora alpha 16
lora dropout 0.05
lora target modules gate_proj, down_proj, up_proj
train on inputs False
add eos token False
group by length False
prompt template alpaca
lr scheduler cosine
warmup steps 100

I would reccomend using a batch size of 4-10, and cutt off length to ≤ 2048 to avoid using vram issues. Load_in_4bit, Normal Float, and bf16. For single 24 gig card.
If training with oobabooga you must edit the "training.py" file in the "oobabooga_windows\text-generation-webui\modules" folder. In line 49 edit standard modules to the modules you would like to target.
If training with alpaca lora use the argument --lora_target_modules when running the train.py command. To load in 4bit you must edit the train file, adding load in 4 bit, bf16, and normal float quant.

原始信息汇总

数据集概述

数据集来源与格式

• 本数据集是基于"psmathurs"的Mini orca数据集的修改版本，格式化为alpaca格式，且未加密。

适用模型与模块

• 适用于ALPACA LORA模型的模块包括："gate_proj", "down_proj", "up_proj", "q_proj", "v_proj", "k_proj", "o_proj"。
• 多数LORA模型常用模块："q_proj", "v_proj", "k_proj", "o_proj"。
• 取得优异结果的Platypus模型使用模块："gate_proj", "down_proj", "up_proj"。

训练参数建议

• Platypus模型的超参数设置：

  • 批量大小：16
  • 微批量大小：1
  • 训练周期数：1
  • 学习率：4e-4 / 3e-4
  • 截断长度：4096
  • LORA等级：16
  • LORA alpha：16
  • LORA dropout：0.05
  • LORA目标模块：gate_proj, down_proj, up_proj
  • 训练输入：False
  • 添加EOS令牌：False
  • 按长度分组：False
  • 提示模板：alpaca
  • 学习率调度器：cosine
  • 预热步骤：100

• 推荐使用参数：

  • 批量大小：4-10
  • 截断长度：≤ 2048，以避免VRAM问题。
  • 加载方式：4bit, Normal Float, bf16。

训练配置修改

• 使用oobabooga进行训练时，需编辑"oobabooga_windows ext-generation-webuimodules"目录下的"training.py"文件，修改第49行的标准模块为目标模块。
• 使用alpaca lora进行训练时，需在运行train.py命令时添加--lora_target_modules参数。
• 加载4bit需编辑train文件，添加load in 4 bit, bf16, and normal float quant。