Code underlying the publication: Efficient Sequential Neural Network based on Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images

This is the source code of the paper: Patil, S.#, <strong>Dong, Y.#</strong>,*, Farah, H, &amp; Hellendoorn, J. (2024). Efficient Sequential Neural Network Based on Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images (Under Review)<br>How to use the codes<strong>(1) Download tvtLANE Dataset:</strong>You can download this **dataset** from the link in the '**Dataset-Description-v1.2.pdf**' file. BaiduYun：https://pan.baidu.com/s/1lE2CjuFa9OQwLIbi-OomTQ passcodes：tf9xOr Google Drive: https://drive.google.com/drive/folders/1MI5gMDspzuV44lfwzpK6PX0vKuOHUbb_?usp=sharing The **pretrained model** is also provided in the "<strong>/model</strong>" folder, named as 98.48263448061671_RAd_lr0.001_batch70_FocalLoss_poly_alpha0.25_gamma2.0_Attention_UNet_LSTM.pth . <strong>(2) Set up</strong><em>## Requirements</em>PyTorch 0.4.0 Python 3.9 CUDA 8.0 <em>## Preparation</em><em>### Data Preparation</em>The dataset contains 19383 continuous driving scenes image sequences, and 39460 frames of them are labeled. The size of images is 128*256.The training set contains 19096 image sequences. Each 13th and 20th frame in a sequence are labeled, and the image and their labels are in “clips_13(_truth)” and “clips_20(_truth)”. All images are contained in “clips_all”. Sequences in “0313”, “0531” and “0601” subfolders are constructed on TuSimple lane detection dataset, containing scenes in American highway. The four “weadd” folders are added images in rural road in China. The testset has two parts: Testset #1 (270 sequences, each 13th and 20th image is labeled) for testing the overall performance of algorithms. The Testset #2 (12 kinds of hard scenes, all frames are labeled) for testing the robustness of algorithms. To input the data, we provide three index files(train_index, val_index, and test_index). Each row in the index represents for a sequence and its label, including the former 5 input images and the last ground truth (corresponding to the last frame of 5 inputs).The dataset needs to be put into a folder with regards to the location in index files, (i.e., txt files in "./data/". The index files should also be modified add cording to your local computer settings. If you want to use your own data, please refer to the format of our dataset and indexes. <strong>(3) Training</strong>Before training, change the paths including "train_path"(for train_index.txt), "val_path"(for val_index.txt), "pretrained_path" in<strong> config_Att.py</strong> to adapt to your environment. Choose the models (<strong>UNet_ConvLSTM</strong> | SCNN_UNet_ConvLSTM | SCNN_UNet_Attention) as the default one which is also indicated by <em>default='UNet-ConvLSTM'</em> thus you do not need to make change for this. And adjust the arguments such as <em>class weights </em>(now the weights are set to fit the tvtLANE dataset), <em>batch size</em>, <em>learning rate</em>, and <em>epochs</em> in <strong>config_Att.py</strong>. You can also adjust other settings, e.g., optimizer, check in the codes for details.Then simply run: <strong>train.py</strong>. If running successfully, there will be model files saved in the "./model" folder. The validating results will also be printed. <strong>(4) Test</strong>To evaluate the performance of a trained model, please select the trained model or put your own models into the "./model/" folder and change "pretrained_path" in <strong>test.py</strong> according to the local setting, then change "test_path" to the location of <em>test_index.txt</em>, and "save_path" for the saved results.Choose the right model that would be evaluated, and then simply run: <strong>test.py</strong>. The quantitative evaluations of Accuracy, Precision, Recall, and F1 measure would be printed, and the lane detection segmented results will be saved in the "./save/" folder as pictures. AuthorsYongqi Dong ([e-mail]), Sandeep Patil, Haneen Farah, Hans Hellendoorn<br>

本仓库为以下论文的源代码：Patil, S.#、董勇琦(Dong, Y.#)*、Farah H、Hellendoorn, J.（2024年，已投稿待审）：基于时空注意力（Spatial-Temporal Attention）与线性长短期记忆网络（Linear LSTM）的高效时序神经网络——用于多帧图像鲁棒车道线检测。

### 代码使用方法
(1) 下载tvtLANE数据集
本数据集可通过《Dataset-Description-v1.2.pdf》文件内的链接下载。百度网盘链接：https://pan.baidu.com/s/1lE2CjuFa9OQwLIbi-OomTQ 提取码：tf9xOr；谷歌云端硬盘链接：https://drive.google.com/drive/folders/1MI5gMDspzuV44lfwzpK6PX0vKuOHUbb_?usp=sharing。预训练模型（pretrained model）亦存放于`/model`文件夹中，文件名为`98.48263448061671_RAd_lr0.001_batch70_FocalLoss_poly_alpha0.25_gamma2.0_Attention_UNet_LSTM.pth`。

(2) 环境配置与数据准备
#### 依赖环境
PyTorch 0.4.0、Python 3.9、CUDA 8.0。
#### 数据准备
本数据集包含19383段连续驾驶场景图像序列，其中39460帧图像带有标注。图像分辨率为128×256。
训练集包含19096段图像序列。序列中的第13帧与第20帧均带有标注，对应图像与标注分别存放于`clips_13(_truth)`与`clips_20(_truth)`文件夹中；全部图像均存放于`clips_all`文件夹内。
`0313`、`0531`与`0601`子文件夹中的序列基于TuSimple车道线检测数据集构建，包含美国高速公路驾驶场景；4个`weadd`子文件夹则补充了中国乡村道路的驾驶图像。
测试集分为两部分：测试集#1包含270段序列，其第13与第20帧带有标注，用于评估算法的整体性能；测试集#2包含12类困难场景，所有帧均带有标注，用于验证算法的鲁棒性。
为便于数据加载，我们提供了3个索引文件：train_index、val_index与test_index。索引文件中每一行对应一段序列及其标注，包含前5帧输入图像与对应的最后一帧真值标签（即5帧输入的最后一帧的标注）。
数据集需按照索引文件（即`./data/`下的txt文件）中记录的路径存放，同时需根据本地环境修改索引文件中的路径配置。若需使用自定义数据集，请参考本数据集与索引文件的格式进行构建。

(3) 模型训练
训练前，请先修改`config_Att.py`文件中的路径参数：包括训练集索引路径`train_path`（对应train_index.txt）、验证集索引路径`val_path`（对应val_index.txt）与预训练模型路径`pretrained_path`，以适配本地运行环境。
该文件中已通过`default='UNet-ConvLSTM'`指定默认模型为`UNet_ConvLSTM`（可选模型包括`UNet_ConvLSTM`、`SCNN_UNet_ConvLSTM`与`SCNN_UNet_Attention`），无需额外修改。同时可在`config_Att.py`中调整各类训练参数，例如类别权重（class weights，当前权重适配tvtLANE数据集）、批次大小（batch size）、学习率（learning rate）与训练轮次（epochs）。其他可调整的参数（如优化器）请参考源代码文件。
完成配置后，直接运行`train.py`即可启动训练。若训练成功，训练得到的模型文件将保存至`./model`文件夹中，同时会输出验证集评估结果。

(4) 模型测试
若需评估已训练模型的性能，请将训练好的模型或自定义模型放入`./model/`文件夹中，随后修改`test.py`文件中的预训练模型路径`pretrained_path`以适配本地环境，再将测试集索引路径`test_path`修改为`test_index.txt`的存放位置，并设置结果保存路径`save_path`。
选择待评估的模型后，直接运行`test.py`即可启动测试。程序将输出准确率（Accuracy）、精确率（Precision）、召回率（Recall）与F1值等量化评估指标，车道线检测的分割结果将以图片形式保存至`./save/`文件夹中。

### 作者信息
董勇琦（[e-mail]）、Sandeep Patil、Haneen Farah、Hans Hellendoorn