Merging Pre-trained Large Language Models

Model merging is a technique that combines two or more LLMs into a single model. The various approaches to merging LLMs can be broadly categorized based on the primary mechanism they employ for combining the models. These categories include weight-based merging, knowledge-based merging, architecture-based merging, and optimization-based merging.

模型合并是一种将两个或多个大语言模型（LLMs）整合为单一模型的技术。合并大语言模型的各类方法可根据其采用的核心整合机制大致分类，这些类别包括基于权重的合并、基于知识的合并、基于架构的合并以及基于优化的合并。