Research on the application of LLaVA model based on Q-LoRA fine-tuning in medical teaching

The Augmented Reality Large Language Model Medical Teaching System integrates Augmented Reality with LLaVA-Med, a medical multimodal large language model based on LLaVA and specifically designed for biomedical applications, employing QLoRA to advance medical education. Deployed on resource-constrained AR devices, such as INMO Air2 glasses, ARLMT overlays real-time visual annotations and textual feedback on medical scenarios to create an immersive and interactive learning environment. Key advancements include a 66\% reduction in memory footprint (from 15.2 GB to 5.1 GB) through QLoRA, enabling efficient operation without compromising performance, and an average response time of 1.009 seconds across various medical imaging categories, surpassing the GPT-4 baseline in both speed and accuracy. The system achieves 98.3% diagnostic accuracy, demonstrating its reliability in real-time applications. By combining visual and textual elements, ARLMT enhances comprehension of complex medical concepts, providing a scalable, real-time solution that bridges technological innovation and pedagogical needs in medical training.增强现实大语言模型医学教学系统（Augmented Reality Large Language Model Medical Teaching System，以下简称ARLMT）将增强现实（Augmented Reality，AR）与LLaVA-Med相结合——后者是一款基于LLaVA、专为生物医学场景设计的医疗多模态大语言模型，通过QLoRA技术赋能医学教育。该系统部署于资源受限的AR设备（如INMO Air2智能眼镜）之上，可在医学场景中叠加实时视觉标注与文本反馈，打造沉浸式交互式学习环境。其核心优化亮点包括：通过QLoRA技术将内存占用量降低66%（从15.2 GB降至5.1 GB），在不牺牲性能的前提下实现高效运行；同时在各类医学影像类别下的平均响应时间仅为1.009秒，在速度与精度两方面均优于GPT-4基准模型。该系统的诊断准确率可达98.3%，证实了其在实时应用场景中的可靠性。通过融合视觉与文本元素，ARLMT可提升学习者对复杂医学概念的理解能力，提供一套可扩展的实时解决方案，搭建起医学培训领域技术创新与教学需求之间的桥梁。