Visualising Multi-Sensor Predictions from a Rice Disease Classifier

The Microsoft Rice Disease Classification Challenge introduced a dataset comprising RGB and RGNiR (RG-Near-infra-Red) images. This second image type increased the difficulty of the challenge such that all of the winning models worked with RGB only. In this challenge we applied a res2next50 encoder that was first pre-trained with self-supervised learning through the SwAV algorithm, to represent each RGB and their corresponding RGNIR images with the same weights. The encoder was then fine-tuned and self-distilled to classify the images which produced a public test set score of 0.228678639, and a private score of 0.183386940. K-fold cross-validation was not used for this challenge result. To better understand the impact of self-supervised pre-training on the problem of classifying each image type, we apply t-distributed Stochastic Neighbour Embedding (t-SNE) on the logits (predictions before applying softmax). We show how this method graphically provides some of the value of a confusion matrix, by locating some incorrect predictions. We then render the visualisation by overlaying the raw images in each data point, and note that to this model, the RGNIR images do not appear to be inherently more difficult to categorise. We make no comparisons through sweeps, RGB-only models or RGNIR-only models. This is left to future work.

微软水稻病害分类挑战赛（Microsoft Rice Disease Classification Challenge）发布了一套包含RGB图像与RGNiR（RG-Near-infra-Red）图像的数据集。该第二类图像拉高了本次挑战赛的难度门槛，以至于所有获奖模型均仅采用RGB图像完成参赛任务。本次挑战赛中，我们采用了经SwAV算法完成自监督预训练的res2next50编码器，以相同权重对每张RGB图像及其对应的RGNiR图像进行特征表征。随后对该编码器进行微调与自蒸馏以实现图像分类任务，最终取得公开测试集得分0.228678639、私有测试集得分0.183386940。本次挑战赛的结果未使用K折交叉验证。为进一步探究自监督预训练对不同图像类型分类任务的影响，我们将t分布随机邻域嵌入（t-distributed Stochastic Neighbour Embedding，t-SNE）应用于logits（Softmax激活前的预测值）。研究表明，该方法可通过定位部分错误预测结果，以可视化形式部分实现混淆矩阵的分析价值。随后我们通过在每个数据点叠加原始图像的方式生成可视化结果，并发现对于该模型而言，RGNiR图像本质上并非天生更难分类。本次研究未通过超参数扫描、仅使用RGB的模型或仅使用RGNiR的模型开展对比实验，相关对比工作将留待后续研究完成。