Deep learning extraction of overlapping galactic compact binaries and EMRI gravitational waves by coupling of ensemble separation and recursive reasoning methods

Among the rich spectrum of GW sources, galactic compact binaries (GCBs) and extreme mass-ratio inspirals (EMRIs) stand out as crucial targets for space-based detectors. GCBs pose challenges in signal extraction due to their overlapping nature. This paper introduces a deep learning framework designed to separate overlapping GCB and EMRI GW signals. The framework employs a two-stage approach: initially, we consider the mixed GCB waveforms as an ensemble, and an ensemble separation method is utilized to separate the mixed GCBs and EMRI signals; subsequently, a recursive reasoning process is applied to further isolate individual GCB signals from the mixed GCB ensemble. We demonstrate the model's robust performance across varying signal-to-noise ratios (SNRs) and overlapping signal counts. The framework exhibits high separation fidelity, particularly for the ensemble separation stage, with overlap metrics exceeding 0.998 under the same parameter ranges of the training set, thereby ensuring accurate signal extraction for subsequent recursive reasoning. For the recursive reasoning process, we have empirically demonstrated that the deep learning framework is capable of effectively separating mixed GCB GW signals even when the frequency differences between them are near or marginally below the frequency resolution limit. We have also observed that the proposed framework exhibits generalization capabilities when applied to GW strain data characterized by lower SNR ranges and larger numbers of mixed GCB signals.