Clinical-T5: Large Language Models Built Using MIMIC Clinical Text

Recent advances in scaling large language models (LLMs) has resulted in
significant improvements over a number of natural language processing
benchmarks. There has been some work to pretrain these language models over
clinical text. These works demonstrate that training a language model using
masked language modeling (MLM) on clinical notes is an effective technique for
boosting performance on downstream tasks. All of these previous works use
decoder-only architectures. We train 4 different clinical T5 models on the
union of MIMIC-III and IV notes. Two of the models are initialized from
previous T5-models (T5-base and SciFive). We additionally train a T5-Base and
T5-Large model from scratch. These models should not be distributed to non-
credentialed users. Research has shown that these language models have the
potential to leak sensitive information. Due to this potential risk, we
release the model weights under PhysioNet credentialed access.