harvardairobotics/FairVLMed

--- license: cc-by-nc-nd-4.0 task_categories: - image-classification - image-to-text modality: - image - text language: - en tags: - medical - ophthalmology - fairness - vision-language - glaucoma - fundus - clinical-notes pretty_name: Harvard-FairVLMed size_categories: - 10K<n<100K --- # Dataset Card: Harvard-FairVLMed ## Dataset Summary Harvard-FairVLMed is the **first fair vision-language medical dataset** designed for studying fairness in medical vision-language (VL) foundation models. It contains 10,000 SLO fundus images paired with de-identified clinical notes and comprehensive demographic annotations, enabling in-depth fairness analysis across four protected attributes: race, gender, ethnicity, and preferred language. This dataset was introduced at CVPR 2024: [FairCLIP: Harnessing Fairness in Vision-Language Learning](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf). ## Dataset Details ### Dataset Description | Field | Value | |-----------------|-------| | **Institution** | Department of Ophthalmology, Harvard Medical School | | **Task** | Glaucoma classification (vision-language) | | **Modality** | Scanning Laser Ophthalmoscopy (SLO) fundus images + clinical notes | | **Scale** | 10,000 patients, 10,000 samples | | **Splits** | 7,000 train / 1,000 validation / 2,000 test | | **License** | [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) | - **Curated by:** Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, Yi Fang, Mengyu Wang - **License:** [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) — non-commercial research only - **Paper:** [CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf) - **Contact:** [e-mail], [e-mail] ### Data Fields Each subject is stored as a `.npz` file (`data_xxxxx.npz`) containing: | Field | Description | |-----------------|-------------| | `slo_fundus` | Scanning Laser Ophthalmoscopy (SLO) fundus image | | `md` | Visual field mean deviation | | `tds` | 52 visual field total deviation values | | `age` | Patient age | | `gender` | `0` = Female, `1` = Male | | `race` | `0` = Asian, `1` = Black, `2` = White | | `ethnicity` | `0` = Non-Hispanic, `1` = Hispanic, `-1` = Unknown | | `language` | `0` = English, `1` = Spanish, `2` = Other, `-1` = Unknown | | `maritalstatus` | `0` = Married/Partnered, `1` = Single, `2` = Divorced, `3` = Widowed, `4` = Legally Separated, `-1` = Unknown | | `glaucoma` | `0` = Non-Glaucoma, `1` = Glaucoma | | `note` | Original de-identified clinical note | | `note_extra` | De-identified clinical note with demographic attributes prepended | A metadata overview is provided in `data_summary.csv`. GPT-4-summarized versions of all notes are available in `gpt-4_summarized_notes.csv`. ### Demographics 6 demographic attributes are annotated per patient: age, gender, race, ethnicity, preferred language, and marital status. ## Uses ### Direct Use - Fairness benchmarking for medical vision-language foundation models (e.g., CLIP, BLIP-2) - Analysis of demographic disparities in ophthalmic AI across race, gender, ethnicity, and language - Development and evaluation of bias-mitigation methods for VL models in healthcare - Medical note summarization and multimodal learning research ### Out-of-Scope Use Clinical decisions, patient care, or any commercial application. This dataset shall not be used for clinical decisions at any time. ## Access The "Harvard" designation indicates the dataset originates from the Department of Ophthalmology at Harvard Medical School. It does not imply endorsement, sponsorship, or legal responsibility by Harvard University or Harvard Medical School. ## Citation **BibTeX:** ```bibtex @inproceedings{luo2024fairclip, title={Fairclip: Harnessing fairness in vision-language learning}, author={Luo, Yan and Shi, Min and Khan, Muhammad Osama and Afzal, Muhammad Muneeb and Huang, Hao and Yuan, Shuaihang and Tian, Yu and Song, Luo and Kouhana, Ava and Elze, Tobias and others}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={12289--12301}, year={2024} } ``` **APA:** Luo, Y., Shi, M., Khan, M. O., Afzal, M. M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., Fang, Y., & Wang, M. (2024). FairCLIP: Harnessing Fairness in Vision-Language Learning. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)*, 12289–12301.--- 许可证：CC BY-NC-ND 4.0 任务类别： - 图像分类 - 图像转文本 模态： - 图像 - 文本 语言： - 英语 标签： - 医疗 - 眼科学 - 公平性 - 视觉语言（Vision-Language, VL） - 青光眼 - 眼底 - 临床笔记 展示名称：Harvard-FairVLMed 规模类别： - 10000 < 样本数 < 100000 --- # 数据集卡片：Harvard-FairVLMed ## 数据集概述 Harvard-FairVLMed是**首个用于研究医疗视觉语言（Vision-Language, VL）基础模型公平性的公平视觉语言医疗数据集**。该数据集包含10000张扫描激光检眼镜（Scanning Laser Ophthalmoscopy, SLO）眼底图像，搭配已去标识化的临床笔记与完整的人口统计学标注，支持针对种族、性别、族裔与偏好语言四类受保护属性开展深入的公平性分析。 本数据集于CVPR 2024大会发布：[FairCLIP：驾驭视觉语言学习中的公平性](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf) ## 数据集详情 ### 数据集描述 | 字段 | 取值 | |-----------------|-------| | **机构** | 哈佛医学院眼科学系 | | **任务** | 青光眼分类（视觉语言方向） | | **模态** | 扫描激光检眼镜（SLO）眼底图像 + 临床笔记 | | **规模** | 10000名患者，10000个样本 | | **划分方式** | 7000个训练样本 / 1000个验证样本 / 2000个测试样本 | | **许可证** | [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) | - **整理方**：Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, Yi Fang, Mengyu Wang - **许可证**：[CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) — 仅用于非商业研究 - **相关论文**：[CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf) - **联系方式**：[e-mail], [e-mail] ### 数据字段 每个受试者以`.npz`文件格式存储（命名格式为`data_xxxxx.npz`），其中包含： | 字段 | 描述 | |-----------------|-------------| | `slo_fundus` | 扫描激光检眼镜（SLO）眼底图像 | | `md` | 视野平均偏差（Visual field mean deviation） | | `tds` | 52个视野总偏差值 | | `age` | 患者年龄 | | `gender` | `0` = 女性，`1` = 男性 | | `race` | `0` = 亚裔，`1` = 黑人，`2` = 白人 | | `ethnicity` | `0` = 非西班牙裔，`1` = 西班牙裔，`-1` = 未知 | | `language` | `0` = 英语，`1` = 西班牙语，`2` = 其他，`-1` = 未知 | | `maritalstatus` | `0` = 已婚/有伴侣，`1` = 单身，`2` = 离异，`3` = 丧偶，`4` = 合法分居，`-1` = 未知 | | `glaucoma` | `0` = 非青光眼，`1` = 青光眼 | | `note` | 原始已去标识化的临床笔记 | | `note_extra` | 前置人口统计学属性的已去标识化临床笔记 | 数据集提供元数据总览文件`data_summary.csv`。所有笔记的GPT-4摘要版本可在`gpt-4_summarized_notes.csv`中获取。 ### 人口统计学信息 每位患者均标注了6项人口统计学属性：年龄、性别、种族、族裔、偏好语言与婚姻状况。 ## 使用场景 ### 直接使用场景 - 医疗视觉语言基础模型（如CLIP、BLIP-2）的公平性基准测试 - 分析眼科学AI在种族、性别、族裔与语言维度上的人口统计学差异 - 开发并评估医疗领域视觉语言模型的偏见缓解方法 - 医疗笔记摘要与多模态学习相关研究 ### 禁止使用场景 临床决策、患者护理或任何商业应用。本数据集绝不可用于任何临床决策场景。 ## 使用声明 “Harvard”标识仅表明本数据集源自哈佛医学院眼科学系，并不代表哈佛大学或哈佛医学院的背书、赞助或法律责任。 ## 引用格式 **BibTeX格式：** bibtex @inproceedings{luo2024fairclip, title={Fairclip: Harnessing fairness in vision-language learning}, author={Luo, Yan and Shi, Min and Khan, Muhammad Osama and Afzal, Muhammad Muneeb and Huang, Hao and Yuan, Shuaihang and Tian, Yu and Song, Luo and Kouhana, Ava and Elze, Tobias and others}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={12289--12301}, year={2024} } **APA格式：** Luo, Y., Shi, M., Khan, M. O., Afzal, M. M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., Fang, Y., & Wang, M. (2024). FairCLIP: Harnessing Fairness in Vision-Language Learning. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)*, 12289–12301.