HPAI-BSC/HEART
收藏资源简介：
--- license: cc-by-4.0 task_categories: - visual-question-answering language: - en tags: - medical pretty_name: HEART size_categories: - 10K<n<100K dataset_info: features: - name: dataset dtype: string - name: task dtype: string - name: approach dtype: string - name: sample_type dtype: string - name: source_id dtype: string - name: true_class dtype: int64 - name: index dtype: int64 - name: question dtype: string - name: A dtype: string - name: B dtype: string - name: C dtype: string - name: D dtype: string - name: answer dtype: string - name: fake_option dtype: string - name: true_bbox dtype: string - name: true_class_name dtype: string - name: fake_bboxes dtype: string - name: fake_bboxes_classes dtype: string - name: fake_class dtype: float64 - name: fake_class_name dtype: string - name: caption dtype: string - name: syco_trigger dtype: string - name: answer_color dtype: string - name: image_path dtype: image splits: - name: train num_bytes: 1597468148 num_examples: 24311 download_size: 1156244174 dataset_size: 1597468148 configs: - config_name: default data_files: - split: train path: data/train-* --- # HEART <div align="center"> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/HEART_logo_transparent.png" width="20%" alt="HPAI"/> </div> <hr style="margin: 15px"> <div align="center" style="line-height: 1;"> <a href="https://hpai.bsc.es/" target="_blank" style="margin: 1px;"> <img alt="Web" src="https://img.shields.io/badge/Website-HPAI-8A2BE2" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/HPAI-BSC" target="_blank" style="margin: 1px;"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-HPAI-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://github.com/HPAI-BSC" target="_blank" style="margin: 1px;"> <img alt="GitHub" src="https://img.shields.io/badge/GitHub-HPAI-%23121011.svg?logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> </div> <div align="center" style="line-height: 1;"> <a href="https://www.linkedin.com/company/hpai" target="_blank" style="margin: 1px;"> <img alt="Linkedin" src="https://img.shields.io/badge/Linkedin-HPAI-blue" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://bsky.app/profile/hpai.bsky.social" target="_blank" style="margin: 1px;"> <img alt="BlueSky" src="https://img.shields.io/badge/Bluesky-HPAI-0285FF?logo=bluesky&logoColor=fff" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://linktr.ee/hpai_bsc" target="_blank" style="margin: 1px;"> <img alt="LinkTree" src="https://img.shields.io/badge/Linktree-HPAI-43E55E?style=flat&logo=linktree&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> </div> <div align="center" style="line-height: 1;"> <a href="https://creativecommons.org/licenses/by/4.0/deed.en" target="_blank" style="margin: 1px;"> <img alt="License" src="https://img.shields.io/badge/license-CC%20BY%204.0-green" style="display: inline-block; vertical-align: middle;"/> </a> </div> ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Summary](#dataset-summary) - [Dataset Sources](#dataset-sources) - [Dataset Modalities](#dataset-modalities) - [Dataset Distribution](#dataset-distribution) - [Dataset Creation](#dataset-creation) - [Dataset Structure](#dataset-structure) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) ## Dataset Summary The HEART dataset is composed of multiple splits that differ in the type of injected cues. It includes a *baseline* split with no injected cues and four cue-based splits. Each cue is instantiated in two variants: *assistive*, where the cue is consistent with the GT, and *adversarial*, where the cue supports an incorrect option. Cue types, summarized below, fall into two categories: *prompt-only* cues where the image is unchanged, and *overlay* cues where information is rendered onto the image. - **Baseline.** No cue is added to the original question and available options. These are constructed by shuffling the GT option with wrong alternatives to establish baseline performance. - **Sycophancy (prompt-only).** The prompt asserts a suggested answer, testing whether the model follows the assertion over visual evidence. - **Prompt captions (prompt-only).** A caption-like description is included in the text prompt. In adversarial samples it is constructed to support a distractor label or box. - **Image captions (overlay).** The caption is rendered onto the image as a visual overlay, enabling evaluation of visually embedded textual cues under the same assistive/adversarial setup. - **Legends (overlay; detection only).** Candidate boxes are shown with a legend, mapping box identifiers to labels. The adversarial variant modifies the mapping to support a distractor box. <div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 2px;"> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/baseline_example.png" /> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/sycophantic_example.png" /> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/image_caption_example.png" /> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/legend_example.png" /> </div> ### Dataset Sources Data sources used to generate the HEART dataset: - **[ARCADE](https://www.nature.com/articles/s41597-023-02871-z)**: a dataset consisting of *X-ray angiography* images of coronary arteries. From the original dataset, we use the test partition for the stenosis detection task (300 images), which contains annotations by medical experts marking regions affected by atherosclerotic plaques. - **[Breast-Lesions-USG](https://www.nature.com/articles/s41597-024-02984-z)**: a dataset containing 252 *breast ultrasound* scans from different patients, manually annotated with benign and malignant lesions. - **[DENTEX](https://zenodo.org/records/7812323)**: a dataset composed of *panoramic dental X-rays* collected from three different institutions, with images annotated by experts. We use the subset of 250 fully labeled X-rays for abnormal tooth detection. These include four specific diagnosis categories: caries, deep caries, periapical lesions, and impacted teeth. - **[BRISC2025](https://arxiv.org/pdf/2506.14318)**: a dataset composed of 1,000 (test partition only) annotated *MRI scans* for brain tumor segmentation and classification, covering glioma, meningioma, pituitary tumors, and non-tumorous cases. - **[HyperKvasir](https://www.nature.com/articles/s41597-020-00622-y)**: a multi-class image dataset for *gastrointestinal endoscopy*. Since the segmented class corresponds only to the polyps category (1,000 images), we selected images from several other pathological finding classes to form the non-polyps category (857 images in total). - **[FractAtlas](https://www.nature.com/articles/s41597-023-02432-4)**: a dataset composed of 4,083 annotated *X-ray* images for fracture detection and localization, covering wrist, ankle, hip, shoulder, and other common fracture sites. Only 1434 images are utilized to keep a balance between fractured and non-fractured samples. - **[PALM](https://www.nature.com/articles/s41597-024-02911-2)**: a dataset composed of 1,200 annotated *fundus photographs* for retinal disease classification covering pathological myopia and normal controls and optic disc segmentation. - **[ISIC 2017](https://arxiv.org/abs/1710.05006)**: a dataset composed of 2,750 annotated *dermoscopic images* of skin lesions, out of which 600 belong to the test partition, each paired with a diagnosis. It covers three diagnostic categories: melanoma (117 images), seborrheic keratosis (90 images), and benign nevi (393 images), and includes expert segmentations. ### Dataset Modalities The image illustrates examples from all medical imaging modalities included in the HEART dataset. <p align="center"> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/modalities.png" width="60%"> </p> ### Dataset Distribution For each adversarial subset, we generate 250 samples per configuration: 250 for the assistive and 250 for the adversarial setup. Datasets containing only detection tasks (ARCADE and DENTEX) yield a total of 2,250 samples each, distributed among the baseline, sycophancy, legends, image captions, and prompt captions subsets. Datasets including both classification and detection tasks (BRISC2025, HyperKvasir, FractAtlas, ISIC2017, and PALM) produce 4,000 samples each, as subsets are created for both task types. The Breast-Lesions-USG dataset contains fewer samples (3,811) due to the inability to generate valid false bounding boxes for certain images (*e.g.,* real bounding boxes cover the entire image). Overall, the final **HEART** dataset consists of **24,311 samples**. ## Dataset Creation HEART is created with [BAIT](https://huggingface.co/HPAI-BSC/BAIT), which takes as input an existing dataset with images and appropriate annotations (image-level labels for classification and bounding boxes with category labels for detection) and a configuration file specifying question templates, option sampling rules, enabled cue types, and, when applicable, overlay rendering settings. For each selected task and cue type, BAIT outputs multiple-choice samples containing an image reference (original or rendered), a prompt, a set of answer options, the correct option, and metadata identifying the task. The same process is applied to every sample in the source data: 1) Input loading and validation: each input sample is first validated against missing or corrupted files, ensuring all required metadata (e.g., annotations or labels) are present and well-formed. 2) Candidate option construction: for classification, BAIT forms a set containing the GT label and distractors. For detection, it forms a set containing the GT box and incorrect boxes. When insufficient negative boxes are available, BAIT can generate plausible distractor ("fake") boxes (see subsection A.1) that * Do not overlap with ground truth boxes * Are sampled from regions with similar low-level statistics (e.g., mean/std of pixel intensities) * Maintain realistic size variation relative to GT boxes * Avoid predominantly empty regions 3) Sample instantiation and cue injection: BAIT selects a template and fills placeholders using GT information (assistive) or distractor information (adversarial), then it assembles the final multiple-choice prompt. If a cue type requires an in-image overlay (e.g., captions or legends rendered on the image), it also outputs a rendered copy of the image. ## Dataset Structure HEART is composed of 8 subsets, each presenting the same structure. <div> <img src="https://huggingface.co/datasets/HPAI-BSC/HEART/resolve/main/Dataset Structure.svg" /> </div> ## Considerations for Using the Data ### Licesing Information The dataset is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) ### Citation Information ``` @inproceedings{suarez2026heart, title={HEART Attacks: Healthcare Evaluation of Adversarial RobusTness}, author={Su{\'a}rez-Fern{\'a}ndez, Mart{\'\i}n and Lopez-Cuena, Enrique and Guasch-Mart{\'\i}, Jaume and Garcia-Gasulla, Dario and Arias-Duart, Anna}, booktitle={The 2026 ACM Conference on Fairness, Accountability, and Transparency}, pages={321--351}, year={2026} } ```
The HEART dataset is composed of multiple splits that differ in the type of injected cues. It includes a baseline split with no injected cues and four cue-based splits. Each cue is instantiated in two variants: assistive, where the cue is consistent with the GT, and adversarial, where the cue supports an incorrect option. Cue types, summarized below, fall into two categories: prompt-only cues where the image is unchanged, and overlay cues where information is rendered onto the image. - Baseline: No cue is added to the original question and available options. These are constructed by shuffling the GT option with wrong alternatives to establish baseline performance. - Sycophancy (prompt-only): The prompt asserts a suggested answer, testing whether the model follows the assertion over visual evidence. - Prompt captions (prompt-only): A caption-like description is included in the text prompt. In adversarial samples it is constructed to support a distractor label or box. - Image captions (overlay): The caption is rendered onto the image as a visual overlay, enabling evaluation of visually embedded textual cues under the same assistive/adversarial setup. - Legends (overlay; detection only): Candidate boxes are shown with a legend, mapping box identifiers to labels. The adversarial variant modifies the mapping to support a distractor box. The dataset is built from eight source medical imaging datasets covering modalities such as X-ray angiography, breast ultrasound, panoramic dental X-rays, brain MRI, gastrointestinal endoscopy, fracture X-rays, fundus photography, and dermoscopic images, totaling 24,311 samples. It is designed to evaluate the adversarial robustness of medical visual question answering models.
