counterfactual-av-eval

# Do Audio-Visual Large Language Models Really See and Hear? [Paper](https://huggingface.co/papers/2604.02605) | [Project Page](https://ramaneswaran.github.io/avllm_interpretability/) | [GitHub](https://github.com/ramaneswaran/avllm_interpretability) This dataset is part of the first mechanistic interpretability study of Audio-Visual Large Language Models (AVLLMs). It is designed to analyze how audio and visual features evolve and fuse through different layers of models like Qwen 2.5 Omni. The data specifically supports investigating modality bias and how models handle conflicting information between audio and vision. ## Dataset Features - `video_captions`: Descriptions of the visual content. - `audio_captions`: Descriptions of the audio content. - `video_path`: The video file associated with the sample. - `group_id`: Identifier for grouping related samples. - `counterfactual`: A boolean indicating if the sample contains conflicting audio and visual information (used for bias analysis). - `id`: Unique identifier for the sample. ## Usage The associated GitHub repository provides code for experiments conducted in the paper. ### Logit Lens Experiment ```bash python src/logitlens_experiment.py \ --model_path Qwen/Qwen2.5-Omni-3B \ --video_path assets/02321.mp4 ``` ### Attention Knockout Experiment ```bash python src/attention_knockout_experiment.py \ --model_path Qwen/Qwen2.5-Omni-3B \ --video_path assets/02321.mp4 ``` ## Citation ```bibtex @article{selvakumar2024avllms, title={Do Audio-Visual Large Language Models Really See and Hear?}, author={Selvakumar, Ramaneswaran and Jayakumar, Kaousheik and Sakshi, S and Ghosh, Sreyan and Gao, Ruohan and Manocha, Dinesh}, journal={arXiv preprint}, year={2024} } ```