PlasEval: a framework for comparing and evaluating plasmid binning tools

PlasEval is a tool aimed at evaluating the accuracy and at comparing methods for the problem of plasmid binning. It has two modes: evaluate and compare. In the evaluation mode of PlasEval, a given set of predicted plasmid bins resulting from a plasmid binning method is evaluated against a ground truth set of plasmid bins, yielding three statistics, the precision, the recall and the F1-score. In the comparison mode of PlasEval, given two sets of predicted plasmid bins (either resulting from two plasmid binning tools or from a plasmid binning tool and a ground truth), PlasEval computes a dissimilarity measure that indicates how much both sets of predicted plasmid bins are in agreement. The data shared consists of output files of PlasBin-flow as well as those of 3 other plasmid binning methods, namely, HyAsP, MOB-recon and gplas for 53 test samples. Details about each sample have been provided in the file Ecoli_samples.csv. It also includes the short read assembly files (fasta as well as gfa) and the hybrid assembly files. The mappings of short-read contigs to hybrid contigs for all 53 samples have been provided in the BLAST_output folder. The plasmid bins, including those representing the ground truth, used as input for PlasEval have been provided in the PlasEval_input folder.PlasEval是一款面向质粒分箱任务的工具，旨在评估相关方法的准确性并对比不同质粒分箱方法的性能，其具备两种运行模式：评估模式与对比模式。 在PlasEval的评估模式下，工具会将某一质粒分箱方法生成的预测质粒分箱集合，与基准真值（ground truth）质粒分箱集合进行比对评估，最终输出精确率（precision）、召回率（recall）与F1分数（F1-score）三项统计指标。 在PlasEval的对比模式下，当输入两组预测质粒分箱集合时（可分别来自两款不同的质粒分箱工具，或来自一款质粒分箱工具与基准真值集合），工具会计算一种相异性度量指标，用以表征两组预测质粒分箱集合的一致性程度。 本次共享的数据集包含53个测试样本的输出文件，其中既有PlasBin-flow的输出结果，也涵盖另外三款质粒分箱工具——HyAsP、MOB-recon与gplas的输出结果。每个测试样本的详细信息已在Ecoli_samples.csv文件中给出。数据集还包含短读长组装文件（fasta与gfa格式）以及混合组装文件。所有53个样本的短读长重叠群（contig）与混合组装重叠群的映射关系，均已存放于BLAST_output文件夹中。而用于PlasEval输入的质粒分箱集合（包含代表基准真值的分箱集合），则被存放于PlasEval_input文件夹中。