Fine-Tuned Mistral-7B for Retail Banking Customer Service

**Overview**
This model, "Fine-Tuned Mistral-7B for Retail Banking Customer Service", is a fine-tuned version of the [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), specifically tailored for the Retail Banking domain. It is optimized to answer questions and assist users with various banking transactions. It has been trained using hybrid synthetic data generated using our NLP/NLG technology and our automated Data Labeling (DAL) tools.

The goal of this model is to show that a generic verticalized model makes customization for a final use case much easier. For example, if you are "ACME Bank", you can create your own customized model by using this fine-tuned model and a doing an additional fine-tuning using a small amount of your own data. An overview of this approach can be found at: [From General-Purpose LLMs to Verticalized Enterprise Models](https://www.bitext.com/blog/general-purpose-models-verticalized-enterprise-genai/)

**Intended Use**
- **Recommended applications**: This model is designed to be used as the first step in Bitext’s two-step approach to LLM fine-tuning for the creation of chatbots, virtual assistants and copilots for the Retail Banking domain, providing customers with fast and accurate answers about their banking needs.
- **Out-of-scope**: It should not be used for non-banking related inquiries or for providing advice on medical, legal, or critical safety issues.
- This model represents step one of of our two-step approach to verticalizing GenAI for enterprise use, as we describe in https://www.bitext.com/blog/general-purpose-models-verticalized-enterprise-genai/

**Model Architecture**
The model uses the `MistralForCausalLM` architecture with a `LlamaTokenizer`. It retains the core features of the base model but is enhanced to address the specific needs of retail banking.

**Training Data**
The model was trained using a dataset designed for retail banking interactions, now publicly available on Hugging Face as [bitext/Bitext-retail-banking-llm-chatbot-training-dataset](https://huggingface.co/datasets/bitext/Bitext-retail-banking-llm-chatbot-training-dataset). This dataset covers 26 different intents such as `check_balance`, `transfer_money`, `open_account`, and more, each with around 1000 examples.

The dataset includes:
- 25,545 question/answer pairs
- 4.98 million tokens
- 1224 entity/slot types

Each entry consists of:
- Instruction: User request
- Category: High-level semantic category
- Intent: Specific intent of the user request
- Response: Example response from a virtual assistant

The dataset covers a wide range of banking-related categories such as ACCOUNT, ATM, CARD, CONTACT, FEES, FIND, LOAN, PASSWORD, and TRANSFER, ensuring comprehensive training for handling diverse retail banking queries.

**Training Procedure**

**Hyperparameters**
- **Optimizer**: AdamW
- **Learning Rate**: 0.0002 with a cosine learning rate scheduler
- **Epochs**: 3
- **Batch Size**: 4
- **Gradient Accumulation Steps**: 4
- **Maximum Sequence Length**: 8192 tokens

**Environment**
- **Transformers Version**: 4.43.4
- **Framework**: PyTorch 2.3.1+cu121
- **Tokenizers**: Tokenizers 0.19.1

**Limitations and Bias**
- The model is specifically trained for retail banking and may not yield accurate results outside this field.
- There may be biases in the training data, which could influence the model's responses, particularly in nuanced financial scenarios.

**Ethical Considerations**
Care should be taken to ensure that the model's automated responses do not replace professional human advice where necessary, particularly in complex financial situations.

**Acknowledgments**
This model was developed and trained by Bitext using their proprietary technologies and resources.

**License**
"Mistral-7B-Retail-Banking" is licensed under the Apache License 2.0 by Bitext Innovations International, Inc. This license permits free use, modification, and distribution, but it requires appropriate attribution to Bitext.

For full details, visit [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).

搜集汇总

背景与挑战

背景概述

该数据集是Mistral-7B模型的零售银行客户服务专用微调版本，采用混合合成数据和自动化标注工具训练，覆盖26种银行交易意图。模型基于Apache 2.0许可，适用于构建银行客服聊天机器人等应用。