harvardairobotics/FairVLMed

---
license: cc-by-nc-nd-4.0
task_categories:
- image-classification
- image-to-text
modality:
- image
- text
language:
- en
tags:
- medical
- ophthalmology
- fairness
- vision-language
- glaucoma
- fundus
- clinical-notes
pretty_name: Harvard-FairVLMed
size_categories:
- 10K<n<100K
---

# Dataset Card: Harvard-FairVLMed

## Dataset Summary

Harvard-FairVLMed is the **first fair vision-language medical dataset** designed for studying fairness in medical vision-language (VL) foundation models. It contains 10,000 SLO fundus images paired with de-identified clinical notes and comprehensive demographic annotations, enabling in-depth fairness analysis across four protected attributes: race, gender, ethnicity, and preferred language.

This dataset was introduced at CVPR 2024: [FairCLIP: Harnessing Fairness in Vision-Language Learning](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf).

## Dataset Details

### Dataset Description

| Field | Value |
|-----------------|-------|
| **Institution** | Department of Ophthalmology, Harvard Medical School |
| **Task** | Glaucoma classification (vision-language) |
| **Modality** | Scanning Laser Ophthalmoscopy (SLO) fundus images + clinical notes |
| **Scale** | 10,000 patients, 10,000 samples |
| **Splits** | 7,000 train / 1,000 validation / 2,000 test |
| **License** | [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) |

- **Curated by:** Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, Yi Fang, Mengyu Wang
- **License:** [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) — non-commercial research only
- **Paper:** [CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/papers/Luo_FairCLIP_Harnessing_Fairness_in_Vision-Language_Learning_CVPR_2024_paper.pdf)
- **Contact:** [e-mail], [e-mail]

### Data Fields

Each subject is stored as a `.npz` file (`data_xxxxx.npz`) containing:

| Field | Description |
|-----------------|-------------|
| `slo_fundus` | Scanning Laser Ophthalmoscopy (SLO) fundus image |
| `md` | Visual field mean deviation |
| `tds` | 52 visual field total deviation values |
| `age` | Patient age |
| `gender` | `0` = Female, `1` = Male |
| `race` | `0` = Asian, `1` = Black, `2` = White |
| `ethnicity` | `0` = Non-Hispanic, `1` = Hispanic, `-1` = Unknown |
| `language` | `0` = English, `1` = Spanish, `2` = Other, `-1` = Unknown |
| `maritalstatus` | `0` = Married/Partnered, `1` = Single, `2` = Divorced, `3` = Widowed, `4` = Legally Separated, `-1` = Unknown |
| `glaucoma` | `0` = Non-Glaucoma, `1` = Glaucoma |
| `note` | Original de-identified clinical note |
| `note_extra` | De-identified clinical note with demographic attributes prepended |

A metadata overview is provided in `data_summary.csv`. GPT-4-summarized versions of all notes are available in `gpt-4_summarized_notes.csv`.

### Demographics

6 demographic attributes are annotated per patient: age, gender, race, ethnicity, preferred language, and marital status.

## Uses

### Direct Use

- Fairness benchmarking for medical vision-language foundation models (e.g., CLIP, BLIP-2)
- Analysis of demographic disparities in ophthalmic AI across race, gender, ethnicity, and language
- Development and evaluation of bias-mitigation methods for VL models in healthcare
- Medical note summarization and multimodal learning research

### Out-of-Scope Use

Clinical decisions, patient care, or any commercial application. This dataset shall not be used for clinical decisions at any time.

## Access

The "Harvard" designation indicates the dataset originates from the Department of Ophthalmology at Harvard Medical School. It does not imply endorsement, sponsorship, or legal responsibility by Harvard University or Harvard Medical School.

## Citation

**BibTeX:**

```bibtex
@inproceedings{luo2024fairclip,
title={Fairclip: Harnessing fairness in vision-language learning},
author={Luo, Yan and Shi, Min and Khan, Muhammad Osama and Afzal, Muhammad Muneeb and Huang, Hao and Yuan, Shuaihang and Tian, Yu and Song, Luo and Kouhana, Ava and Elze, Tobias and others},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={12289--12301},
year={2024}
}
```

**APA:**

Luo, Y., Shi, M., Khan, M. O., Afzal, M. M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., Fang, Y., & Wang, M. (2024). FairCLIP: Harnessing Fairness in Vision-Language Learning. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)*, 12289–12301.