AdaptFerm: Bioprocess Monitoring Using FTIR spectroscopy: Insights into Substrate Effects and Domain Adaptation

1. Introduction The AdaptFerm dataset is designed to support the development of a monitoring framework for lactic acid production fermentation using Fourier Transform Infrared (FTIR) spectroscopy. Its primary goal is to facilitate the control strategies for continuous fermentation processes to maximize the lactic acid production. The AdaptFerm encompasses data from two distinct batch fermentation environments: one employing simple sugar (glucose) as the substrate and the other utilizing complex sugars derived from bio-waste. The study focuses on developing accurate predictive models for glucose and lactic acid concentrations, with an emphasis on applying classical machine learning techniques and enhancing domain generalization capabilities. 2. Prediction Model for Different Substrate Environments The chemical composition of substrates are presented in Table 1 [1]. The dataset is utilized to train and test models within the same substrate domain. For instance, data from a single fermentation environment (e.g., glucose substrate) is used for both training and testing phases. The applied machine learning models showed accurate prediction within the same domain [1]. For more details on the methods applied, please refer to the following link: https://doi.org/10.1016/j.heliyon.2024.e38791. In this study, the MIR results correspond to the AdaptFerm dataset. The spectra of the glucose and biowaste hydrolysate fermentation process are presented in Figure 3 and Figure 4. 3. Domain Adaptation The dataset was also used to address the challenge posed by shifts in FTIR data when substrates change. Transitioning from simple sugar (glucose) to complex sugar (bio-waste) causes significant variations in the FTIR spectra, making it difficult for models trained on glucose fermentation data to maintain prediction accuracy in the complex sugar fermentation environment. This results in reduced robustness and performance when applied to out-of-distribution data. To address these challenges, we explore methods that improve the generalization ability and robustness of models in such scenarios without using labels from complex sugar fermentation [2]. It shows the application of machine learning interpretation to find domain invariant features for glucose and lactic acid. For more details on the methods applied, please refer to the following link: https://dx.doi.org/10.2139/ssrn.5012080. The code is available at https://github.com/shl-shawn/ShapFS. 4. Real-World Use Cases 4.1. Regression Task AdaptFerm serves as a benchmark for machine learning model applications in fermentation processes, specifically for predicting glucose and lactic acid concentrations, measured in g/L (grams per liter), while considering issues of out-of-distribution generalization. 4.2. Domain Adaptation Regression Task The dataset is also suitable for evaluating different domain adaptation methods. In particular, the glucose substrate fermentation data can be used as the source domain, while the complex sugar fermentation data from bio-waste serves as the target domain. For semi-supervised domain adaptation approaches, it is recommended to use the initial data points (i.e., those collected at the beginning of the fermentation process) from the target domain, as the dataset is organized chronologically by collection day. These approaches aim to improve the robustness of models by transferring knowledge across domains and mitigating the effects of out-of-distribution data. 4.3. Anomaly Detection The dataset can be used to train anomaly detection models to identify outliers or deviations from normal fermentation behavior. This could be valuable in industrial bioprocessing, where early detection of issues like contamination or process failure is crucial. Techniques like Isolation Forests, One-Class SVM, or Autoencoders could be applied to identify unusual patterns in FTIR spectra. 4.4. Classification Task Although the main task is regression, the dataset could also be used in classification tasks by discretizing the concentrations of glucose and lactic acid into categories (e.g., low, medium, high). This would allow for the application of classification algorithms like Support Vector Machines (SVM), Random Forests, or Neural Networks for predicting the fermentation phase or identifying specific operational conditions. 4.5. Transfer Learning Given the nature of the domain adaptation approach in this dataset, transfer learning models can be explored. Models pre-trained on glucose fermentation data can be fine-tuned on complex sugar fermentation data, enabling quicker model convergence and improved performance in data-scarce environments. 4.6. Multi-Task Learning In a multi-task learning scenario, models could simultaneously predict both glucose and lactic acid concentrations from the same FTIR data. This could help in improving model accuracy by leveraging shared representations across the two tasks. 4.7. Feature Selection The FTIR spectral data contains a large number of features (wavelengths), and feature selection techniques such as Recursive Feature Elimination (RFE), Lasso regression, or mutual information could be applied to identify the most relevant wavelengths for predicting glucose and lactic acid concentrations, improving model performance and interpretability. 5. Dataset Structure and Meta Information The dataset is organized into four Excel files, corresponding to two main fermentation domains (different substrates) and two key process variables: a) Simple Sugar SubstrateThis domain contains data for the fermentation process using glucose as the substrate to produce lactic acid. It includes two files—one for glucose concentrations and one for lactic acid concentrations. Both are measured in g/L. b) Complex Sugar SubstrateThis doman contains data for the fermentation process using bio-waste as the substrate to produce lactic acid. Similar to the previous domain, it includes two files—one for glucose concentrations and one for lactic acid concentrations. Both are measured in g/L. Each file is structured as follows: The first column contains the sample ID, which serves as the timeline of measurements (Sample ID 1 represents the first measurement in the fermentation process). From the second column onwards, the FTIR data is provided, covering the spectral range from 549.6 cm-1 to 3999.6 cm-1 comprising 3,579 features. The final column contains the ground truth data, the chemical measurements of fermentation variables such as glucose and lactic acid concentrations, both measured in g/L. 6. Conclusion The AdaptFerm features FTIR spectra data from two distinct fermentation environments: simple sugar (glucose) and complex sugar (bio-waste). The dataset is designed to be used in regression tasks, including domain adaptation, and can be applied in machine learning model development for fermentation process monitoring, with a focus on enhancing model robustness and handling out-of-distribution data. This dataset provides a valuable resource for exploring domain shift and improving the robustness of machine learning models in bioengineering and fermentation processes. It enables further research into domain generalization techniques and offers a wide range of possibilities for machine learning applications. References [1] Arman Arefi, Barbara Sturm, Majharulislam Babor, Michael Horf, Thomas Hoffmann, Marina Höhne, Kathleen Friedrich, Linda Schroedter, Joachim Venus, Agata Olszewska-Widdrat, Digital model of biochemical reactions in lactic acid bacterial fermentation of simple glucose and biowaste substrates, Heliyon, Volume 10, Issue 19, 2024, e38791, ISSN 2405-8440, DOI: 10.1016/j.heliyon.2024.e38791, https://doi.org/10.1016/j.heliyon.2024.e38791. [2] Majharulislam Babor, Shanghua Liu, Arman Arefi, Agata Olszewska-Widdrat, Barbara Sturm, Joachim Venus, and Marina M.-C. Höhne, Domain-Invariant Monitoring for Lactic Acid Production: Transfer Learning from Glucose to Bio-Waste Using Machine Learning Interpretation. Available at http://dx.doi.org/10.2139/ssrn.5012080.1. 引言 AdaptFerm数据集旨在为基于傅里叶变换红外（Fourier Transform Infrared, FTIR）光谱的乳酸生产发酵监测框架开发提供支撑。其核心目标是助力连续发酵过程的控制策略优化，以实现乳酸产量最大化。该数据集涵盖两种不同分批发酵环境下的数据：一种以单糖（葡萄糖）为底物，另一种使用源自生物废弃物的复合糖。本研究聚焦于开发葡萄糖与乳酸浓度的精准预测模型，重点应用经典机器学习技术并提升模型的域泛化能力。 2. 不同底物环境下的预测模型 底物的化学组成详见表1[1]。该数据集用于在同一底物域内训练与测试模型，例如仅使用单一发酵环境（如葡萄糖底物）的数据完成模型的训练与测试阶段。相关研究表明，所采用的机器学习模型可在同一域内实现精准预测[1]。有关所用方法的详细信息，请参阅以下链接：https://doi.org/10.1016/j.heliyon.2024.e38791。本研究中的中红外（Mid-Infrared, MIR）结果均对应AdaptFerm数据集。葡萄糖与生物废弃物水解液发酵过程的光谱分别如图3与图4所示。 3. 域自适应 该数据集还可用于应对底物更换时FTIR数据发生分布偏移带来的挑战。从单糖（葡萄糖）切换至复合糖（生物废弃物）会导致FTIR光谱发生显著变化，使得在葡萄糖发酵数据上训练的模型难以在复合糖发酵环境中维持预测精度，在应用于分布外数据时，模型的鲁棒性与性能会出现下降。为解决上述问题，本研究探索了在不使用复合糖发酵数据标签的前提下，提升此类场景下模型泛化能力与鲁棒性的方法[2]。该方法借助机器学习可解释性技术挖掘葡萄糖与乳酸的域不变特征。有关所用方法的详细信息，请参阅以下链接：https://dx.doi.org/10.2139/ssrn.5012080。相关代码已开源至https://github.com/shl-shawn/ShapFS。 4. 实际应用场景 4.1. 回归任务 AdaptFerm可作为发酵过程中机器学习模型应用的基准数据集，尤其适用于预测以克每升（g/L）为单位的葡萄糖与乳酸浓度，同时兼顾分布外泛化问题。 4.2. 域自适应回归任务 该数据集同样适用于评估各类域自适应方法。具体而言，可将葡萄糖底物发酵数据作为源域，将源自生物废弃物的复合糖发酵数据作为目标域。对于半监督域自适应方法，建议使用目标域的初始数据点（即发酵过程初期采集的数据），因为数据集按采集日期按时间顺序组织。此类方法旨在通过跨域知识迁移缓解分布外数据带来的负面影响，提升模型鲁棒性。 4.3. 异常检测 该数据集可用于训练异常检测模型，以识别发酵过程中的异常值或偏离正常行为的情况。这在工业生物加工领域具有重要价值，因为早期检测污染或工艺故障等问题至关重要。可采用孤立森林（Isolation Forests）、单类支持向量机（One-Class SVM）或自编码器（Autoencoders）等技术识别FTIR光谱中的异常模式。 4.4. 分类任务 尽管本数据集的核心任务为回归任务，但也可通过将葡萄糖与乳酸浓度离散化为低、中、高等类别，将其应用于分类任务。这使得支持向量机（Support Vector Machines, SVM）、随机森林（Random Forests）或神经网络等分类算法可用于预测发酵阶段或识别特定运行工况。 4.5. 迁移学习 鉴于本数据集的域自适应特性，可探索迁移学习模型的应用。在葡萄糖发酵数据上预训练的模型可在复合糖发酵数据上进行微调，从而加快模型收敛速度，并在数据稀缺的场景下提升模型性能。 4.6. 多任务学习 在多任务学习场景中，模型可基于同一份FTIR数据同时预测葡萄糖与乳酸浓度。通过利用两项任务间的共享表征，可提升模型精度。 4.7. 特征选择 FTIR光谱数据包含大量特征（波长），可采用递归特征消除（Recursive Feature Elimination, RFE）、Lasso回归或互信息等特征选择技术，筛选出与葡萄糖及乳酸浓度预测最相关的波长，提升模型性能与可解释性。 5. 数据集结构与元信息 该数据集分为4个Excel文件，对应两类主要发酵域（不同底物）与两类核心工艺变量： a) 单糖底物域 该域涵盖以葡萄糖为底物生产乳酸的发酵过程数据，包含两个文件：分别对应葡萄糖浓度与乳酸浓度数据，单位均为g/L。 b) 复合糖底物域 该域涵盖以生物废弃物为底物生产乳酸的发酵过程数据，与前述域类似，包含两个文件：分别对应葡萄糖浓度与乳酸浓度数据，单位均为g/L。 各文件的结构如下： 第一列为样本ID，作为测量的时间轴（样本ID 1代表发酵过程中的首次测量）。 从第二列起为FTIR数据，覆盖549.6 cm⁻¹至3999.6 cm⁻¹的光谱范围，共包含3579个特征。 最后一列为真实标签（ground truth）数据，即葡萄糖与乳酸浓度等发酵变量的化学测量值，单位均为g/L。 6. 结论 AdaptFerm数据集包含两类不同发酵环境的FTIR光谱数据：单糖（葡萄糖）与复合糖（生物废弃物）。该数据集专为包括域自适应在内的回归任务设计，可用于发酵过程监测的机器学习模型开发，重点提升模型鲁棒性并处理分布外数据。本数据集为探索域偏移问题、提升生物工程与发酵过程中机器学习模型的鲁棒性提供了宝贵资源，为域泛化技术的后续研究提供了支撑，并为机器学习的各类应用提供了广阔空间。 参考文献 [1] Arman Arefi, Barbara Sturm, Majharulislam Babor, Michael Horf, Thomas Hoffmann, Marina Höhne, Kathleen Friedrich, Linda Schroedter, Joachim Venus, Agata Olszewska-Widdrat. Digital model of biochemical reactions in lactic acid bacterial fermentation of simple glucose and biowaste substrates[J]. Heliyon, 2024, Volume 10, Issue 19, e38791. ISSN 2405-8440. DOI: 10.1016/j.heliyon.2024.e38791. https://doi.org/10.1016/j.heliyon.2024.e38791. [2] Majharulislam Babor, Shanghua Liu, Arman Arefi, Agata Olszewska-Widdrat, Barbara Sturm, Joachim Venus, Marina M.-C. Höhne. Domain-Invariant Monitoring for Lactic Acid Production: Transfer Learning from Glucose to Bio-Waste Using Machine Learning Interpretation[EB/OL]. http://dx.doi.org/10.2139/ssrn.5012080, 2024.