EchoPT: A Pretrained Transformer Architecture for Predicting 2D In-Air Sonar Images in Mobile Robotics

EchoGPT

This folder contains the supplementary data and code for the submission "EchoPT: A Pretrained Transformer Architecture for Predicting 2D In-Air Sonar Images in Mobile Robotics" to the NeurIPS 2024 conference.

Dependencies

Matlab 2024a or higher is needed with the following toolboxes:

Image Processing Toolbox

Parallel Computing Toolbox

Deep Learning Toolbox

Signal Processing Toolbox

System Identification Toolbox

Data

Simulation

This folder contains the saved sonar images (energyscapes) as well as the motion data for each frame from a particular simulation run (LongRun1). These are saved into batches and saved as compressed .mat files within the \DataCalculated\SimulationData\LongRun1\Raw folder.

Trained model

The model used in the experiments of this submission is saved in \DataCalculated\Networks as a Matlab dlnetwork object. There is a trained (with weights) and an untrained version available.

Code

Data pre-processing

To extract the data from the batch .mat files into seperate frames a script preprocessEchoPT.m is available in the main folder. This will save these individual frames once again to individual .mat files into a folder \DataCalculated\ESSequences by default.

Training

To train the model a Matlab script trainEchoPT.m is provided.

Evaluation

To generate the figures of the submission as well as additional outputs such as GIFs three different evaluation scripts are available:

evaluateEchoPT.m: Evaluate the trained model on the dataset.

evaluateEchoPT_ComparisonAccFlow_ARLoop.m: Evaluate the trained model on the dataset and compare it to acoustic flow with autoregressive prediction.

evaluateEchoPT_ComparisonAccFlow_NoAR.m: Evaluate the trained model on the dataset and compare it to acoustic flow without autoregressive prediction.

Source

Within the folder \Source all additional Matlab functions and classes can be found.

Open-Source libraries included in this project

Progress bar by HyunGwang Cho (link)