Uncertainty and Novelty in Machine Learning

Uncertainty and novelty are inherent in machine learning, especially as new information is encountered and the hypothesis set’s best model is to be determined given the current information. Ideally, we could answer the following: what are the types of uncertainty and novelty that a predictor could encounter and how do we measure them, how does uncertainty and novelty effect the information perceived from observations, and how can a predictor be evaluated when learning a such phenomena.
This work answers these questions through both theory and application. We provide a Bayesian evaluation framework for subjective tasks where different sources of uncertainty are considered and the truth itself is uncertain. We introduce an abstraction of novelty that is then further developed in terms of information theory and algorithms.
This formalizes the concept of identifiable information that arises from the language used to express the relationship between distinct states. Through the computation of the indicator function, model identifiability and sample complexity are defined and their properties are described for different data-generating processes, ranging from deterministic to ergodic stationary stochastic processes. This demonstrates identifying information in finite steps to asymptotic statistics and PAC-learning, where we recover identification within finite observations at the cost of uncertainty and error.
We explore the practical evaluation of novelty detection and adaptation with new benchmarks in handwriting recognition and human activity recognition.

不确定性与新颖性是机器学习（Machine Learning）与生俱来的固有属性，尤其当预测器遭遇新信息、且需基于当前信息确定假设空间中的最优模型时，这一属性更为凸显。理想情况下，我们可解答如下问题：预测器可能遭遇的不确定性与新颖性有哪些类型？我们应如何对其进行量化衡量？不确定性与新颖性会如何影响从观测数据中获取的信息？针对这类现象开展学习时，应如何对预测器进行评估？

本研究从理论与应用两个层面解答了上述问题。针对存在多源不确定性且真值本身亦存在不确定性的主观任务，我们提出了一套贝叶斯（Bayesian）评估框架。我们首次给出了新颖性的抽象定义，并基于信息论与算法视角对其进行了系统化拓展。

该框架将可识别信息（identifiable information）这一概念形式化——该信息源于用于表达不同状态间关联关系的表述语言。通过指示函数（indicator function）的计算，我们定义了模型可辨识性与样本复杂度，并针对从确定性过程到遍历平稳随机过程的各类数据生成过程，阐明了二者的性质特征。该研究将可识别信息的推导从有限步拓展至渐近统计与大概率近似正确学习（PAC-learning）场景，证明了在有限观测样本内可实现可辨识性，但其代价是引入不确定性与误差。

我们基于手写识别与人体活动识别领域的全新基准数据集，对新颖性检测与自适应方法开展了实际评估。