Confusion matrix for custom F-Net classifier.

The study’s primary objectives encompass the following: (i) To implement the object detection of ovarian follicles using you only look once (YOLO)v8 and subsequently segment the identified follicles using a hybrid fuzzy c-means-based active contour technique. (ii) To extract statistical features and evaluate the effectiveness of both machine learning (ML) and deep learning (DL) classifiers in detecting polycystic ovary syndrome (PCOS). The research involved a two different dataset in which dataset1 comprising both normal (N = 50) and PCOS (N = 50) subjects, dataset 2 consists of 100 normal and 100 PCOS affected subjects for classification. The YOLOv8 method was employed for follicle detection, whereas statistical features were derived using Gray-level co-occurrence matrices (GLCM). For PCOS classification, various ML models such as Random Forest (RF), k- star, and stochastic gradient descent (SGD) were employed. Additionally, pre-trained models such as MobileNet, ResNet152V2, and DenseNet121 and Vision transformer were applied for the categorization of PCOS and healthy controls. Furthermore, a custom model named Follicles Net (F-Net) was developed to enhance the performance and accuracy in PCOS classification. Remarkably, the F-Net model outperformed among all ML and DL classifiers, achieving an impressive classification accuracy of 95% for dataset1 and 97.5% for dataset2 respectively in detecting PCOS. Consequently, the custom F-Net model holds significant potential as an effective automated diagnostic tool for distinguishing between normal and PCOS.