Not so weak-PICO: Leveraging weak supervision for Participants, Interventions, and Outcomes recognition for systematic review automation

Objective: PICO (Participants, Interventions, Comparators, Outcomes)
analysis is vital but time-consuming for conducting systematic reviews
(SRs). Supervised machine learning can help fully automate it, but a lack
of large annotated corpora limits the quality of automated PICO
recognition systems. The largest currently available PICO corpus is
manually annotated, which is an approach that is often too expensive for
the scientific community to apply. Depending on the specific SR question,
PICO criteria are extended to PICOC (C-Context), PICOT (T-timeframe), and
PIBOSO (B-Background, S-Study design, O-Other) meaning the static
hand-labelled corpora need to undergo costly re-annotation as per the
downstream requirements. We aim to test the feasibility of designing a
weak supervision system to extract these entities without hand-labelled
data. Methodology: We decompose PICO spans into its constituent entities
and re-purpose multiple medical and non-medical ontologies and
expert-generated rules to obtain multiple noisy labels for these entities.
These labels obtained using several sources are then aggregated using
simple majority voting and generative modelling approaches. The resulting
programmatic labels are used as weak signals to train a weakly-supervised
discriminative model and observe performance changes. We explore mistakes
in the currently available PICO corpus that could have led to inaccurate
evaluation of several automation methods. Results: We present Weak-PICO, a
weakly-supervised PICO entity recognition approach using medical and
non-medical ontologies, dictionaries and expert-generated rules. Our
approach does not use hand-labelled data. Conclusion: Weak supervision
using weak-PICO for PICO entity recognition has encouraging results, and
the approach can potentially extend to more clinical entities readily.