Geo-CLIP-For-Street-View

Folder Contents Description<b>code.zip</b><br>This zip file contains all the code files for the experiment. After extracting, you will have access to the scripts, functions, and source code necessary to run the experiment. For a detailed explanation of the code structure and functionality, please refer to the <b>README-code.md</b> file.<b>data.zip</b><br>This zip file includes all the data files used in the experiment. It contains input data, experimental datasets, and other files related to the experiment. For a detailed explanation of the data and how to use it, please refer to the <b>README-data.md</b> file.Code overview（1）Baselines pretraining - documented in the `/CaseStudy1/base_line_pretrain.ipynb` and `/CaseStudy2/base_line_pretrain.ipynb` files<br>（2） Baselines training and evaluation - documented in the `/CaseStudy1/base_line_train.ipynb` and `/CaseStudy2/base_line_train.ipynb` files<br>（3） Ablation experiments pretraining - documented in the `/CaseStudy1/ablation_experiment_pretrain.ipynb` and `/CaseStudy2/ablation_experiment_pretrain.ipynb` files<br>（4） Ablation experiments training and evaluation - documented in the `/CaseStudy1/ablation_experiment_train.ipynb` and `/CaseStudy2/ablation_experiment_train.ipynb` files<br>（5） t-SNE Visualization for semantic knowledge between geographical locations (Base on case study 1)- documented in the `/CaseStudy1/utils/Visualization/t-SNE.py` file<br>（6）GradCAM++ for street view imagery (Base on case study 1)- documented in the `/CaseStudy1/utils/Visualization/cam_main.py` file<br>（7）Using SHAP to analyse the importance of viusal knowledge in case study 1- documented in the `/CaseStudy1/utils/Visualization/SHAP.py` file<br>（8） Using integrated gradient to analyse the importance of viusal knowledge in case study 2- documented in the `/CaseStudy2/utils/IntegratedGradient/main.py` fileData OverviewThe <b>data</b> directory contains all the datasets and related files used in the experiment.Case study 1: Urban village classification<br>```CaseStudy1/├── data/| ├── train_img.npy # the npy file converted from the original street view images of the training set, shape: (n_train, 4, 3, 256, 256)| ├── train_feat.npy # the visual knowledge of the training set, shape: (n_train, 128)| ├── train_svid.npy # the street view image IDs of the training set, shape: (n_train,)| ├── train_y.npy # the label of the training set, shape: (n_train,)| ├── val_img.npy # the npy file converted from the original street view images of the validation set, shape: (n_val, 4, 3, 256, 256)| ├── val_feat.npy: # the visual knowledge of the validation set, shape: (n_val, 128)| ├── val_svid.npy # the street view image IDs of the validation set, shape: (n_val,)| ├── val_y.npy # the label of the validation set, shape: (n_val,)| ├── dist.npy # the distance matrix, shape: (n, n)| ├── svids.npy # the street view image IDs corresponding to the distance matrix, shape: (n,)<br>with: n: the number of samples n_train: the number of training samples n_val: the number of validation samples n_train : n_val = 7 : 3```<br>Case study 2: Urban mobility pattern prediction<br>``` CaseStudy2/├── data/ | ├── pretrain/| ├── train_img.npy # the npy file converted from the original street view images of the training set, shape: (n_train, 3, 256, 256)| ├── train_feat.npy # the visual knowledge of the training set, shape: (n_train, 30)| ├── train_svid.npy # the street view image IDs of the training set, shape: (n_train,)| ├── val_img.npy # the npy file converted from the original street view images of the validation set, shape: (n_val, 3, 256, 256)| ├── val_feat.npy: # the visual knowledge of the validation set, shape: (n_val, 30)| ├── val_svid.npy # the street view image IDs of the validation set, shape: (n_val,)| ├── dist.npy # the distance matrix, shape: (n, n)| ├── svids.npy # the street view image IDs corresponding to the distance matrix, shape: (n,)| ├── train/| ├── feats/ | ├── 1.npy # the npy file represents the visual knowledge corresponding to the region with ID 1， shape (num_SVI_1, 30)| ├── 2.npy # the npy file represents the visual knowledge corresponding to the region with ID 2， shape (num_SVI_2, 30)| ├── 3.npy # the npy file represents the visual knowledge corresponding to the region with ID 3， shape (num_SVI_3, 30)| ├── imgs/ | ├── 1.npy # the npy file represents the street view imagery corresponding to the region with ID 1， shape (num_SVI_1, 3, 256, 256)| ├── 2.npy # the npy file represents the street view imagery corresponding to the region with ID 2， shape (num_SVI_2, 3, 256, 256)| ├── 3.npy # the npy file represents the street view imagery corresponding to the region with ID 3， shape (num_SVI_3, 3, 256, 256)| ├── fliter/ | ├── train_flow.npy # the taxi flow count of the training set, shape: (n_region_train,5)| ├── train_id.npy # the regions of the training set, shape: (n_region_train,)| ├── val_flow.npy # the taxi flow count of the validation set, shape: (n_region_val,5)| ├── val_id.npy # the regions of the validation set, shape: (n_region_val,)<br>with: n: the number of image samples n_train: the number of training samples n_val: the number of validation samples n_train : n_val = 7 : 3 n_region_train: the number of regions in the training set n_region_val: the number of regions in the validation set n_region_train : n_region_val = 7 : 3<br> num_SVI_1: the number of street view images collected in the region with id 1 num_SVI_2: the number of street view images collected in the region with id 2 num_SVI_3: the number of street view images collected in the region with id 3<br>```For detailed descriptions of the data, its format, and how to use it in the experiment, please refer to the <b>README-data.md</b> file.<br><br><br>