figshare_submit.xlsx

Recently there has been considerable excitement surrounding the promising realization of high-spin Kitaev material, such as the quasi-2D compound CrI3 and CrGeTe3. However, the stability of quantum spin liquids (QSL) against single ion anisotropy (SIA) in these materials and the global quantum phase diagram of the extended spin-3/2 Kitaev model with finite SIA remain unclear. In this study, we perform large-scale density matrix renormalization group (DMRG) to explore the quantum phase diagram of the generalized spin-3/2 Kitaev-Heisenberg (K-H) model accompanied with SIA Ac. In the Ac = 0 limit, the spin-3/2 K-H model exhibits a quantum phase diagram similar to that of a spin-1/2 system, including two QSLs around antiferromagnetic and ferromagnetic Kitaev models. For models with finite Ac, we map out the quantum phase diagram around two Kitaev points and observe distinct types of in-plane vortex orders developed from these two QSL phases. Interestingly, series of nearly degenerate vortex configurations are discovered in each vortex phases. Using linear spin-wave theory, we demonstrate that these vortex configurations can be understood as a consequence of the quantum correction on a continuous family of degenerate classical states.