llama

本项目基于Meta发布的可商用大模型Llama-2开发，是中文LLaMA&Alpaca大模型的第二期项目，开源了中文LLaMA-2基座模型和Alpaca-2指令精调大模型。这些模型在原版Llama-2的基础上扩充并优化了中文词表，使用了大规模中文数据进行增量预训练，进一步提升了中文基础语义和指令理解能力，相比一代相关模型获得了显著性能提升。相关模型支持FlashAttention-2训练。标准版模型支持4K上下文长度，长上下文版模型支持16K、64k上下文长度。RLHF系列模型为标准版模型基础上进行人类偏好对齐精调，相比标准版模型在正确价值观体现方面获得了显著性能提升。

This project is developed based on Meta's commercially available large language model Llama-2, and constitutes the second-phase project of the Chinese LLaMA & Alpaca large language model series. We have open-sourced the Chinese LLaMA-2 base model and Alpaca-2 instruction-tuned large language model. Building upon the original Llama-2, these models expanded and optimized the Chinese vocabulary, and conducted incremental pre-training on large-scale Chinese datasets, thereby further enhancing their basic Chinese semantic understanding and instruction following capabilities, and achieving significant performance improvements compared to the first-generation related models. All relevant models support training with FlashAttention-2. The standard-version models support a 4K context window, while the long-context variants support 16K and 64K context windows. The RLHF-series models are fine-tuned for human preference alignment based on the standard-version models, and have achieved significant performance improvements over the standard-version models in terms of embodying correct values.

搜集汇总

背景与挑战

背景概述

该数据集是中文LLaMA&Alpaca大模型的第二期项目，包含基于Llama-2开发的中文基座模型和指令精调模型，优化了中文词表并支持多种上下文长度和训练技术。