MERFISH measurements in the mouse ileum

Spatial transcriptomics protocols based on in situ sequencing or
multiplexed RNA fluorescent hybridization can reveal detailed tissue
organization. However, distinguishing the boundaries of individual cells
in such data is challenging, and can hamper downstream analysis. Current
methods generally approximate cells positions using nuclei stains. We
describe a segmentation method, Baysor, which optimizes 2D or 3D cell
boundaries considering joint likelihood of transcriptional composition and
cell morphology. While Baysor can take into account segmentation
based on co-stains, it can also perform segmentation based on the detected
transcripts alone. To evaluate performance, we extend MERFISH to
incorporate immuno-staining of cell boundaries. Using this and other
benchmarks we show that Baysor segmentation can in some cases
nearly double the number of cells, while reducing segmentation artifacts.
Importantly, we demonstrate that Baysor performs well on data
acquired using five different protocols, making it a useful general tool
for analysis of imaging-based spatial transcriptomics.