hfl rbt3

# This is a re-trained 3-layer RoBERTa-wwm-ext model.

## Chinese BERT with Whole Word Masking
For further accelerating Chinese natural language processing, we provide **Chinese pre-trained BERT with Whole Word Masking**.

Example
Original Sentence we use a language model to predict the probability of the next word.
MLM we use a language [M] to [M] ##di ##ct the pro [M] ##bility of the next word .
Whole word masking we use a language [M] to [M] [M] [M] the [M] [M] [M] of the next word .
N-gram masking we use a [M] [M] to [M] [M] [M] the [M] [M] [M] [M] [M] next word .
MLM as correction we use a text system to ca ##lc ##ulate the po ##si ##bility of the next word .
Except for the new pre-training task, we also incorporate the following techniques.

Whole Word Masking (WWM)
N-gram masking
Sentence-Order Prediction (SOP)
Note that our MacBERT can be directly replaced with the original BERT as there is no differences in the main neural architecture.

For more technical details, please check our paper: Revisiting Pre-trained Models for Chinese Natural Language Processing

# 本模型为经重新训练的3层RoBERTa-wwm-ext模型。
## 带全词掩码（Whole Word Masking）的中文BERT
为进一步推进中文自然语言处理研究，我们发布了**带全词掩码（Whole Word Masking）的中文预训练BERT**。

### 示例
原句 we use a language model to predict the probability of the next word.
掩码语言模型（Masked Language Model, MLM） we use a language [M] to [M] ##di ##ct the pro [M] ##bility of the next word .
全词掩码（Whole Word Masking, WWM） we use a language [M] to [M] [M] [M] the [M] [M] [M] of the next word .
N元掩码（N-gram masking） we use a [M] [M] to [M] [M] [M] the [M] [M] [M] [M] [M] next word .
用于纠错的掩码语言模型（MLM as correction） we use a text system to ca ##lc ##ulate the po ##si ##bility of the next word .

除新增预训练任务外，我们还集成了以下技术：

全词掩码（Whole Word Masking, WWM）
N元掩码（N-gram masking）
句序预测（Sentence-Order Prediction, SOP）

需注意，由于核心神经架构无差异，我们的MacBERT可直接替代原生BERT使用。

如需了解更多技术细节，请参阅我们的论文：《重新审视中文自然语言处理的预训练模型》

搜集汇总

背景与挑战

背景概述

hfl rbt3是一个基于RoBERTa-wwm-ext的3层重训练模型，专注于中文自然语言处理，采用了全词掩码和N-gram掩码等技术。该模型可直接替代原始BERT，适用于中文文本处理任务。