Vocalization data and scripts to model reindeer rut activity using on-animal acoustic recorders and machine learning

# Vocalization data and scripts to model reindeer rut activity using on-animal acoustic recorders and machine learning \***If the user wishes to just analyze the data, simply download the compressed in the root file*** \****During the Data upload process, the data and code had to be uploaded separately. Consequently, to place the back into software folders, simply download the corresponding compressed data folder and place the uncompressed data folder within the root folder of the specific folder it corresponds to (e.g., the 2-Rut Analysis (Data) folder into the 2-Rut Analysis and rename the 2-Rut Analysis (Data) folder to simply ‘data’ so that the software can find and pull data from said folder) **0 - File_Annotation** This file is separated into 4 subfolders, each similar to the other. The two SVL folders contain files that are easily imported and read by Sonic Visualizer. The other two are the same, but they are in CSV format in case a user wishes to train another machine-learning network using our annotations. The corresponding audio can be found in /Sample (with a small subset)/1-Training_Folders: Verification data/Data/Audio. There are 134 files and the 2020 files are 2.26 hours long, while the 210 files are 8 hours long. 0.5-Sample-Training-and-predicting example This folder contains a small subset of data in case the users wish to test the code on their system. The code is set up in a duplicate manner to that found in the two following folders, just with a much less intensive audio subset (in case the user's computer has limited computation power). The ‘.ipnyb’ are the only two files that need to be interacted with and they can be opened with Jupyter Notebook or imported into Google Colab. Begin with the Training. ipynb file and follow the prompt to train a simple machine-learning network. The process should take about 10-15 minutes, depending on the power of your computer. Then once that initial process is complete, open the Predicting_Fine_Tuning.ipynb file in the notebook and again follow the prompts, this process should take about 10 minutes as well. Afterwards, SVL files will be created, with the network's predictions. These files, along with their respective .WAV files can then be imported into Sonicvisualizer to inspect its predictions. Within the /Data folder, there are a number of subfolders. /Annotations is where the binary annotations for the .WAV files are read from in order to train the networks. /Audio is where the individual .WAV files are stored and read from. /DataFiles are the training and testing .txt files are read from. These lists are read by the script to indicate which annotations and .WAV files the network should either process or predict upon. These files, need to keep the same title as they are currently labelled and the names of the files within need to be exactly the same as the annotation and .WAV files, without their file types. /Model_Ouput is where the trained networks will be saved (as a .hdf5 file, and later, to use the predicting script, this file must be transferred to the root folder, e.g., that containing the Predicting_Fine_Tuning.ipynb file for instance) and the .SVL files or the network-predicted sonic visualizer layers files, which are imported by the user to inspect the network's predictions. /Pickled_Data and /Saved are metadata files that contain script/data generated by the script, the user does not need to be concerned with the contents of these files **1-Training_Folders/Verification data** This folder is identical to the previous one, only that it contains all the annotation and .WAV files used to train our papers network. Note that while the /Audio folder contains 134 audio files, only a subset of these were actually used to train the network, (34 audio files) due to hardware limitations. The full set of audio files corresponds to those found in the 0 - File_Annotation folder, should the user wish to train their network with a more powerful computer, with a larger annotation dataset. Consequently, the TestingFiles and TrainingFiles .txt files correspond to the subset of audio and annotation used to train our final network. The list of 12 files within the TestingFiles.txt corresponds to the 12 audio files used to verify the performance of our networks, as described in our paper. The Verification info subfolder includes network predictions (Both in .SVL and .SV file formats [the audio for these files can be found in the aforementioned /Data/Audio folder. To inspect them yourself, simply import them into Sonic Visualizer. The .xlsx file corresponds to our inspection of these files and our count of TP, FP and FN files and our calculations for Recall rate, Precision, F1 and Accuracy. The two first tabs are the performance of a couple of preliminary networks, and the “Final Set” tab is the calculations reported within our paper. **2-Rut Analysis** Again, this file is set up in a duplicate manner to the previous two, but in this one, the subfolders within the /Data folder are empty, as such, I will leave it to the user to analyze what files they want, the audio and annotation files are organized as follows. Additionally, the training side of this folder, as in this folder, it is meant to simply be the predicting folder, whereas the above folder is meant to be one where the user trains their network. If the user trained a network in the above folder, and wishes to use their weights to predict upon files with this folder, the user should copy and paste their .hdf5 network file into this folder, in a manner duplicate to that mentioned above. -The audio files are contained within the /Sample (with a small subset)/2-Rut Analysis/Day_Audio_segments, here, the files are organized by the individual. These files are all eight hours long, so that each day is easily divided into three sections. Only a few of the audio files are less than 8 hours, as they are when the reindeer were put into the mating pen, or their recorders unfortunately failed. -And the testing files are found within the /Sample (with a small subset)/2-Rut Analysis/Network Predictions/Test_Files subfolders. Again, organized by the individual. Should the user wish to predict some of the audio files, simply place the appropriate .WAV files within the above-mentioned/Audio folder and the appropriate TestingFiles.txt files within the /DataFiles folder. If the user wishes to only predict a couple of files, they will have to edit the TestingFiles.txt file with the appropriate files they wish to predict. If the user wants to view our network's predictions, choose the appropriate .WAV from the location above and find the .SVL from the file below, and place them in SonicVisualizer. -The annotations files are found within the Sample (with a small subset)/2-Rut Analysis/Network Predictions/Original unprocessed SVL Files/Prediction SVL subfolders, again, these are organized by the individual. These were the original predictions as made by our network and remain unprocessed by us. The user can also import CSV files into SonicVisualizer as well. These are the audibly and visually verified prediction annotations produced by the network, as mentioned in our paper. -These annotations can be found in the /Sample (with a small subset)/2-Rut Analysis/Network Predictions/Processed CSV FIles (1/3 day segments) subfolder. In these ones, FP predictions were removed, and FN predictions added, when possible, as it is likely some vocalizations were missed. The /Sample (with small subset)/2-Rut Analysis/Network Predictions/Day Segment Descriptions (along with original FL designations and recording notes) subfolder contains notes on the process as we made the 8-hour segments, along with their original file designations and notes as they were inspected. The /Sample (with small subset)/2-Rut Analysis/Network Predictions/Prediction .XLS Day Segments **AND** /Sample (with small subset)/2-Rut Analysis/Network Predictions/Prediction .XLS Amalgamations - folders contain Excel sheets that were used to tabulate and calculate our proportional data. The ‘Day segments’ folder contains individual day files for each individual as we had recordings for them. Each day had 3 segments and we used these forms to describe their hourly proportional activity across the entire day. The ‘Day’ data from these files was then transferred to the ‘Amalgamations’ folder and the Excel sheets there, as they correspond to the individual, so that we could describe the reindeer activity as it changed over time, from hour to hour of each day, and from day to day. This file has tabs to describe their activity in terms of just days, 8-hour daily segments, and then hourly segments. This was done early on in the process as a form of data exploration. Finally, the /Users/alex_boucher/Desktop/Sample (with small subset)/2-Rut Analysis/Network Predictions/Rut Activity Analysis (R Analysis) include the R code used to analyze our data. In this file, our final master Excel file can be found, and the R code is streamlined for the user to analyze our data as well. To look at our master Excel file, refer to: Boucher et al. (2023) - Reindeer Rut Activity Trends/2-Rut Analysis/Network Predictions/Rut Activity Analysis (R Analysis)/MasterRuttingActivity.xlsx **Notes** -Refer to Boucher_A-MLReindeer/1-Training_folders/TrainingSets.xlsx and Boucher_A-MLReindeer/1-Training_folders/TestSets.xlsx to see a list of files used to train the algorithms - for both, I used the half set - for the absence segments, there were 18218 ‘no grunt’ spectrograms and 8506 ‘grunt’ spectrograms - I augmented the grunt spectrograms by a factor of 2 for a total of 17012 grunt segments and 18218 - the test vs train annotation sets was set 16:37, or roughly a 30:70 test to train split - down-sampled the no_grunt total to 17025 to balance the totals - total sample of 34037 segments -To train/test the algorithms, the code was run on an M1 Pro 16-inch MacBook Pro with 16 GBS of RAM using Apple metal architecture -Training times - 10 Epoch - Training: ~600 seconds per Epoch, Finetuning: ~1850 - total = ~24500 seconds, 408 minutes, 6.8 hours; 50 Epoch - Training: ~600 seconds per Epoch, Finetuning: ~1850 - total = ~122500 seconds, 2041 minutes, 34.3 hours -Transfer learning - used ResNet152V2 for transfer learning - [https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html](https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html) -We had an individual whose recordings were muffled, hence we did not use this individual for this study. If you reference SD46, this is a remnant from before we stopped considering this individual and should be disregarded. The Individuals, as identified in our paper, were done so as follows: GPS39 = 2019 - 5.5 GPS43 = 2019 - 2.5 SD24 = 2020 - 1.5 SD39 = 2020 - 4.5 SD42 = 2020 - 3.5 - 1 SD44 = 2020 - 3.5 - 3 SD47 = 2020 - 3.5 - 2 **Copyright Information** Note that most of this code was adapted from that supplied within the below listed study - to use code in further studies, contact Emmanuel Dufourq for permission - code repository can be found at Passive acoustic monitoring of animal populations with transfer learning - Emmanuel Dufourq, Carly Datist, Ruben Foquet, Ian Durbach - Ecological Informatics Volume 70, Sept 2022, 101688 - [https://www.sciencedirect.com/science/article/pii/S1574954122001388](https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sciencedirect.com%2Fscience%2Farticle%2Fpii%2FS1574954122001388&data=05%7C02%7Crobert.weladji%40concordia.ca%7C0ba30fb07884487c536508dc63ad0787%7C5569f185d22f4e139850ce5b1abcd2e8%7C0%7C0%7C638494842081715868%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l7OKmhIFZFBT2OuVTAR0fQmi7NI74rNulFKafOEG6Z0%3D&reserved=0) Code used was the ‘Hainnon Gibbon - 25 sample - Fine tuning set’ Below is the copyright information copied from the original code license MIT License: Copyright (c) 2022 Emmanuel Dufourq Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

# 利用佩戴式声学记录仪与机器学习建模驯鹿发情期行为的发声数据集与脚本

**若仅需分析数据，仅需下载根目录下的压缩文件即可**

**在数据上传流程中，数据集与代码需分开上传。因此，若需将数据放回软件文件夹中，仅需下载对应的数据压缩包，将解压后的数据文件夹放置于其对应专属文件夹的根目录下即可。例如，将“2-Rut Analysis (Data)”文件夹放入“2-Rut Analysis”文件夹，并将其重命名为“data”，以便软件能够检索并读取该文件夹中的数据**

## 0 - 文件标注（File_Annotation）
本文件夹包含4个结构相似的子文件夹。其中两个SVL格式文件夹的文件可直接由Sonic Visualizer导入读取。剩余两个文件夹采用CSV格式，方便用户使用我们的标注训练其他机器学习网络。配套音频文件可在`/Sample（含小样本子集）/1-Training_Folders: Verification data/Data/Audio`中获取。该目录下共有134个音频文件：2020年的音频总时长为2.26小时，2021年的音频总时长为8小时。

## 0.5 - 样本训练与预测示例（Sample-Training-and-predicting example）
该文件夹包含小样本数据子集，供用户在本地设备上测试代码使用。其代码结构与后续两个文件夹完全一致，但音频子集规模更小，适配计算性能有限的设备。仅需使用两个`.ipynb`文件即可完成交互，可通过Jupyter Notebook打开或导入至Google Colab中运行。首先运行`Training.ipynb`文件，按照提示训练一个简易机器学习网络，该过程耗时约10-15分钟，具体取决于设备性能。完成初始训练后，打开`Predicting_Fine_Tuning.ipynb`文件并按照提示操作，该过程同样耗时约10分钟。运行结束后将生成SVL格式文件，内含网络预测结果。可将这些文件与对应的WAV文件一同导入Sonic Visualizer，以查看模型预测效果。

在`/Data`目录下包含多个子文件夹：
- `/Annotations`：存放用于训练网络的WAV文件二进制标注文件
- `/Audio`：存储并读取单个WAV音频文件的目录
- `/DataFiles`：存放训练与测试用的TXT列表文件，脚本将通过这些列表指定网络需要处理或预测的标注文件与WAV文件。这些文件需保留原有文件名，且内部列出的文件名需与对应标注文件和WAV文件完全一致（不含扩展名）
- `/Model_Output`：用于保存训练好的网络模型（保存为`.hdf5`格式文件。若需使用预测脚本，需将该文件转移至对应根目录，例如包含`Predicting_Fine_Tuning.ipynb`的文件夹），同时存放网络生成的SVL格式文件（即Sonic Visualizer可导入的模型预测层文件），供用户查看模型预测结果
- `/Pickled_Data`与`/Saved`：存放脚本生成的元数据文件，用户无需关注其具体内容

## 1 - 训练文件夹/验证数据（1-Training_Folders/Verification data）
该文件夹与上述文件夹结构一致，仅包含用于训练论文中所用网络的全部标注与WAV文件。需注意：尽管`/Audio`目录下共有134个音频文件，但受硬件限制，实际仅使用其中34个音频文件用于网络训练。若用户使用性能更强的设备，可使用0 - File_Annotation文件夹中的全部音频文件与标注数据集训练网络。`TestingFiles.txt`与`TrainingFiles.txt`中的列表对应了最终训练论文模型所用的音频与标注子集。`TestingFiles.txt`中的12个文件对应了论文中用于验证网络性能的12个音频文件。

该目录下的`Verification info`子文件夹包含网络预测结果（格式为SVL与SV文件，对应的音频文件可在上述`/Data/Audio`目录中获取。用户可将其导入Sonic Visualizer自行查看效果）。其中的`.xlsx`文件记录了我们对这些文件的评估结果，包括真阳性（TP）、假阳性（FP）、假阴性（FN）样本计数，以及召回率、精确率、F1值与准确率的计算结果。前两个工作表为初步网络的性能结果，“Final Set”工作表为论文中报告的最终计算结果。

## 2 - 发情期行为分析（2-Rut Analysis）
该文件夹的代码结构与前两个文件夹一致，但`/Data`目录下的子文件夹均为空，用户可自行选择所需文件进行分析。音频与标注文件的组织方式如下：
本文件夹的训练模块仅作为预测模块使用，而前一个文件夹则用于用户训练网络。若用户已在前一个文件夹中完成网络训练，并希望使用训练好的权重在本文件夹中进行预测，需将训练得到的`.hdf5`网络模型文件按照前述方式复制至本文件夹中。
- 音频文件存放于`/Sample（含小样本子集）/2-Rut Analysis/Day_Audio_segments`，按个体进行组织。所有音频文件时长均为8小时，方便将单日数据划分为三个片段。仅少量音频文件时长不足8小时，对应驯鹿被放入交配圈或记录仪故障的情况。
- 测试文件存放于`/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Test_Files`子文件夹，同样按个体进行组织。若用户希望对部分音频文件进行预测，只需将对应的WAV文件放入上述`/Audio`目录，并将对应的`TestingFiles.txt`文件放入`/DataFiles`目录。若仅需预测少量文件，需手动编辑`TestingFiles.txt`文件，添加待预测的文件名。若用户希望查看我们的网络预测结果，可从上述音频文件目录中选取对应WAV文件，并从下述目录中获取对应的SVL文件，一同导入Sonic Visualizer。
- 原始标注文件存放于`/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Original unprocessed SVL Files/Prediction SVL`子文件夹，同样按个体组织。这些为网络生成的原始预测结果，未经我们人工处理。用户也可将CSV格式文件导入Sonic Visualizer，这些文件为网络生成的经听觉与视觉验证的预测标注，对应论文中提及的结果。
- 经处理的标注文件可在`/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Processed CSV Files（1/3 day segments）`子文件夹中获取。该目录下的文件已移除假阳性预测，并尽可能补全假阴性预测（因可能存在未被检测到的发声）。
- `/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Day Segment Descriptions (along with original FL designations and recording notes)`子文件夹包含8小时音频片段的处理说明，以及原始文件标识与人工检视记录。
- `/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Prediction .XLS Day Segments`与`/Sample（含小样本子集）/2-Rut Analysis/Network Predictions/Prediction .XLS Amalgamations`文件夹包含用于统计与计算比例数据的Excel表格。“Day segments”文件夹包含每个个体的单日数据文件，每个单日划分为三个片段，用于记录单日每小时的行为占比。“Day”目录下的数据将被汇总至“Amalgamations”目录下的Excel表格中，这些表格按个体组织，用于描述驯鹿行为随时间（每日的每一小时、每日之间）的变化情况。该文件包含多个工作表，分别按单日、8小时每日片段、每小时片段展示行为分布，用于研究初期的数据探索。
- 最后，`/Users/alex_boucher/Desktop/Sample (with small subset)/2-Rut Analysis/Network Predictions/Rut Activity Analysis (R Analysis)`目录包含用于数据分析的R代码。该目录下可找到我们的最终主Excel文件，R代码已进行优化，方便用户复刻数据分析流程。如需查看我们的主Excel文件，请参考：Boucher等人（2023）- Reindeer Rut Activity Trends/2-Rut Analysis/Network Predictions/Rut Activity Analysis (R Analysis)/MasterRuttingActivity.xlsx

## 注意事项（Notes）
- 请参考`Boucher_A-MLReindeer/1-Training_folders/TrainingSets.xlsx`与`Boucher_A-MLReindeer/1-Training_folders/TestSets.xlsx`查看用于训练算法的文件列表。本次训练与测试均使用一半数据集：对于非发声片段，共有18218个“无哼叫”语谱图，8506个“有哼叫”语谱图。我们将“有哼叫”语谱图的数据扩增2倍，最终得到17012个“有哼叫”片段与18218个“无哼叫”片段。测试集与训练集的划分比例约为30:70（16:37）。我们将“无哼叫”样本下采样至17025个以平衡数据集，最终总样本量为34037个片段。
- 本次训练与测试基于搭载Apple Metal架构的16英寸MacBook Pro（M1 Pro芯片，16GB RAM）完成。训练耗时：10个Epoch时，每个Epoch训练耗时约600秒，微调耗时约1850秒，总耗时约24500秒（408分钟，6.8小时）；50个Epoch时，每个Epoch训练耗时约600秒，微调耗时约1850秒，总耗时约122500秒（2041分钟，34.3小时）。
- 本次训练采用迁移学习，使用的模型为ResNet152V2，详见论文：[He等人，2016，CVPR](https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html)。
- 有一只个体的录音被遮挡，因此未纳入本研究。若提及SD46，该标识为停用该个体前的残留内容，请勿参考。论文中标识的个体对应关系如下：
GPS39 = 2019 - 5.5
GPS43 = 2019 - 2.5
SD24 = 2020 - 1.5
SD39 = 2020 - 4.5
SD42 = 2020 - 3.5 - 1
SD44 = 2020 - 3.5 - 3
SD47 = 2020 - 3.5 - 2

## 版权信息（Copyright Information）
请注意，本项目大部分代码改编自下述研究提供的源码。若需在后续研究中使用该代码，请联系Emmanuel Dufourq获取授权。代码仓库源自论文《Passive acoustic monitoring of animal populations with transfer learning》（Emmanuel Dufourq, Carly Datist, Ruben Foquet, Ian Durbach - Ecological Informatics Volume 70, Sept 2022, 101688），链接：[https://www.sciencedirect.com/science/article/pii/S1574954122001388](https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sciencedirect.com%2Fscience%2Farticle%2Fpii%2FS1574954122001388&data=05%7C02%7Crobert.weladji%40concordia.ca%7C0ba30fb07884487c536508dc63ad0787%7C5569f185d22f4e139850ce5b1abcd2e8%7C0%7C0%7C638494842081715868%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l7OKmhIFZFBT2OuVTAR0fQmi7NI74rNulFKafOEG6Z0%3D&reserved=0)。本次使用的代码为`Hainnon Gibbon - 25 sample - Fine tuning set`。

下述为原代码的MIT许可证版权声明：
> MIT许可证
> 版权所有（c）2022 Emmanuel Dufourq
> 特此免费向任何获取本软件及其相关文档文件（以下简称“软件”）的个人授予许可，允许不受限制地使用本软件，包括但不限于使用、复制、修改、合并、发布、分发、再许可和/或销售本软件的副本，并允许向其提供本软件的人员在遵守以下条件的前提下使用本软件：
> 上述版权声明与本许可声明应包含在本软件的所有副本或主要部分中。