Semi-naive Bayesian classification

The success and popularity of naive Bayes has led to a field of research exploring algorithms that seek to retain its numerous strengths while reducing error by alleviating the attribute interdependence problem. This thesis builds upon this promising field of research, contributing a systematic survey and several novel and effective techniques. It starts with a study of the strengths and weaknesses of previous semi-naive Bayesian methods, providing a taxonomy of them and comparative analysis of their features. Twelve key semi-naive Bayesian methods are benchmarked using error analysis based on the bias-variance decomposition, probabilistic prediction analysis based on the quadratic loss function and training and classification time analysis on sixty natural domains from the UCI Machine Learning Repository. Results for logistic regression and LibSVM, a popular SVM implementation, are also presented to provide a baseline for comparison. In analyzing results of these experiments, we offer general recommendations for selection between semi-naive Bayesian methods based on the characteristics of the application to which they are applied. This comparative study supports previous findings ofstrong performance from Averaged One-Dependence Estimators (AODE), which significantly reduces naive Bayes’ error with modest training and classification time overheads. Backward Sequential Elimination is an effective wrapper technique to identify and repair harmful interdependencies, and has been profitably applied to naive Bayes. It is therefore surprising that its straightforward application to AODE has previously proved ineffective. In response to this observation, this thesis explores novel variants of this strategy leading to effective techniques. These eliminate child attributes from within the constituent One-Dependence Estimators, thereby significantly improving AODE’s prediction accuracy. However, due to repeated accuracy evaluation of attribute subsets on AODE, these elimination techniques have very high training time complexity. In response to this drawback, a new type of semi-naive Bayesian operation, Subsumption Resolution (SR), is proposed. It efficiently identifies pairs of attribute values such that one is a generalization of the other and deletes the generalization at classification time. This adjustment is proved to be theoretically correct for such an interdependence relationship. The thesis demonstrates experimentally that SR can in practice significantly v improve both classification accuracy and the precision of conditional probability estimates. When applied to AODE, SR achieves classification and probability estimation accuracy competitive to state-of-the-art semi-naive Bayesian methods without undue time complexity and may prove desirable over a considerable range of classification tasks. In addition, SR is suited to incremental and semi-supervised learning. This thesis also explores circumstances under which elimination of near-generalizations proves beneficial.