QKVQA_train

# QKVQA: Question-Focused Filtering for Knowledge-based VQA Retrieval This is the official implementation of [QKVQA: Question-Focused Filtering for Knowledge-based VQA Retrieval] ## 🛠️ Requirements 1. Create conda environment ```bash conda create -n QKVQA python=3.9.19 conda activate QKVQA ``` 2. Install the required packages ```bash pip install -r requirements.txt pip install flash-attn --no-build-isolation ``` ## Data Infoseek You can use this [link](https://ailb-web.ing.unimore.it/publicfiles/drive/reflectiva/data_infoseek.zip) to download the evaluation data for Infoseek. To download the VQA samples in InfoSeek: * [test.csv](https://drive.google.com/file/d/1cSG_dVuao9lKZy8vaUDWEo7mIHowjUeE/view?usp=drive_link) * [InfoSeek's 100K Knowledge Base ](https://drive.google.com/file/d/1cIbKtYryD7XBAw0tjrrCvMCJC2rIzLM5/view?usp=drive_link) ## Data Encyclopedic-VQA The images used for evaluation can be extracted from this [zip file](https://ailb-web.ing.unimore.it/publicfiles/drive/reflectiva/evqa_inference_images.zip). To download the VQA samples and knowledge base in E-VQA: * [test.csv](https://storage.googleapis.com/encyclopedic-vqa/test.csv) * [Encylopedic-VQA's 2M Knowledge Base](https://storage.googleapis.com/encyclopedic-vqa/encyclopedic_kb_wiki.zip) And for iNaturalist 2021: * [val_id2name](https://drive.google.com/file/d/1cYzo4qewPABFuoMhpME4j2DWAA_Y-l2L/view?usp=drive_link) The images used for train: * [train.csv](https://drive.google.com/file/d/13BZZAserLlqKT_RHq4sX5NAIu7Ii77eE/view?usp=drive_link) * [iNaturalist 2021](https://github.com/visipedia/inat_comp/tree/master/2021) ( You should change the image name to image_id with [train_id2name](https://drive.google.com/file/d/1cUP0sWtI4z7whH9V5FOvqfJ0LTxZLOd9/view?usp=drive_link) for the direct connect to the dataset_image_id in vqa sample ) * [Google Landmarks Dataset V2](https://github.com/cvdfoundation/google-landmark) ## Wiki Images Regarding the image data of entity webpages in the knowledge base, we re-crawled the images from Wikipedia to ensure completeness. The entity information, image URLs, and the relative paths for storing the images are recorded in 13 CSV files. These CSV files are provided to facilitate the downloading of knowledge base images: - [Wiki Images CSVs](https://drive.google.com/file/d/1__laja2XMKA-J3oBT7EFdLxLCEGfpw4r/view?usp=drive_link) For each CSV file, a corresponding folder needs to be created to download all associated images. The image URLs and their storage paths can be found in the image_URL and img_path columns of the CSV files. After downloading, the directory structure of the wiki images should be as follows: <pre>. ├── full │ └── wiki_image_split │ ├── wiki_entity_image_1 │ ├── wiki_entity_image_2 │ ├── wiki_entity_image_3 │ ├── wiki_entity_image_4 │ ... │ └── wiki_entity_image_13 └── output ├── wiki_image_url_part_1_processed.csv ├── wiki_image_url_part_2_processed.csv ├── wiki_image_url_part_3_processed.csv ... └── wiki_image_url_part_13_processed.csv</pre> ## Faiss Index - [E-VQA KB Summary Faiss Index](https://drive.google.com/file/d/1FPx1EOTjcx8zXPoPD9fBQ6cbucF0fL_D/view?usp=drive_link) - [InfoSeek KB Summary Faiss Index](https://drive.google.com/file/d/1CE-SyPNKEx6fT09LXoMh4ckmh7jfzG_E/view?usp=drive_link) ## Please replace the relevant path variables in the following files with your own paths: - `utils/utils.py` - `dataset/dataset_utils.py` ## QFF Training - [Hard_Neg Tranning Samples For Reranker](https://drive.google.com/file/d/1X9sOZV5jSielgszfIcncvcoCtDumHkBD/view?usp=drive_link) - [train.csv](https://modelscope.cn/datasets/leaffffff/QKVQA_train) To train the multimodal fusion reranker, run the bash script after changing the necessary configurations. ```bash bash scripts/training_our_acc_question_focused.sh ``` Here, you can download the QFF model weights: `modelscope download --model leaffffff/QKVQA_QFF` ## Script Details The training_our_acc_question_focused.sh script is used to fine-tune the multimodal fusion reranker module with specific parameters: --`reranker-model-name`: Name of the Reranker model. The default value for this item is `qformer_IT2IT_reranker`, which corresponds to the multimodal fusion reranker of QKVQA. --`dataset_name`: The name of dataset used for training. In our work, the dataset is `E-VQA`. --`num-epochs`: Number of epochs for training. In our work, the reranker is trained for 1 epochs. --`num-workers`: Number of worker threads for data loading. --`learning-rate`: Learning rate for the optimizer. The default value is `1e-5`. --`batch-size`: Number of samples per batch during training. The default value is `6`. --`neg-num`: The number of negative pairs included in the each of the training samples. In our work, each sample has 15 corresponding negative pairs. --`save_frequency`: How many epochs between each model checkpoint save. The default value is `0.25`. --`train_file`: Path to the training data file. The training file should be the same format as provided by E-VQA. --`knowledge_base_file`: Path to the knowledge base file in JSON format. The format should be the same with that of the E-VQA. --`negative_db_file`: Path to the hard negative sampled database file used for training. --`wiki_img_csv_dir`: The folder path containing the CSV file that records information related to Wiki images. --`wiki_img_path_prefix`: The path to the parent folder containing the `full/wiki_image_split` folder, which holds all the wiki images from different subsets. --`save-training`: Flag to save the training progress. --`wandb`: Whether to use wandb for training process logging; set the value to 1 if used, otherwise 0. Remember to set your wandb API key in the shell using `WANDB_API_KEY` if you want to use wandb. ## Afterwards, you can run the following scripts: - For experiments on the E-VQA dataset using Qwen2.5-VL-7B, execute: `test/qkvqa_evqa_qwen.py` - For experiments on the InfoSeek dataset using Qwen2.5-VL-7B, execute: `test/qkvqa_infoseek_qwen.py` Once an experiment is completed, the retrieval results can be reused for generating answers with other models. For example: - Run `test/qkvqa_evqa_llama.py` for experiments on E-VQA using LLaMA. - Run `test/qkvqa_infoseek_llama.py` for experiments on InfoSeek using LLaMA. Again, remember to update the path variables in the running scripts to your own paths. ## 💖 Acknowledgements Thanks to the code of [OMGM](https://github.com/ChaoLinAViy/OMGM), [LAVIS](https://github.com/salesforce/LAVIS/tree/main) and [LLaVA](https://github.com/haotian-liu/LLaVA/tree/main/) and data of [ReflectiVA](https://github.com/aimagelab/ReflectiVA), [EchoSight](https://github.com/Go2Heart/EchoSight), [Encyclopedic-VQA](https://github.com/google-research/google-research/tree/master/encyclopedic_vqa) and [InfoSeek](https://github.com/open-vision-language/infoseek).