Active Evaluation Software for Selection of Ground Truth Labels

This software repository contains a python package Aegis (Active Evaluator Germane Interactive Selector) package that allows us to evaluate machine learning systems's performance (according to a metric such as accuracy) by adaptively sampling trials to label from an unlabeled test set to minimize the number of labels needed. This includes sample (public) data as well as a simulation script that tests different label-selecting strategies on already labelled test sets. This software is configured so that users can add their own data and system outputs to test evaluation.

本软件仓库包含Python包Aegis（Active Evaluator Germane Interactive Selector，主动评估关联交互式选择器），可通过从无标注测试集中自适应采样待标注样本，在最小化所需标注的样本数量的前提下，基于准确率等评估指标评估机器学习系统的性能。本仓库附带公开示例数据，以及可在已标注测试集上测试多种标签选择策略的仿真脚本。该软件支持用户自行添加自有数据与系统输出结果，以开展自定义评估实验。