saillab/VinDR-CXR-PSF
收藏资源简介：
--- license: cc-by-4.0 task_categories: - visual-question-answering language: - en tags: - medical - chest-x-ray - vqa - paraphrase-sensitivity - robustness - radiology pretty_name: VinDR-CXR Paraphrase Sensitivity Failure Dataset size_categories: - 10K<n<100K source_datasets: - faizan711/VinDR-CXR-VQA --- # VinDR-CXR-PSF: Paraphrase Sensitivity Failure Evaluation Dataset A benchmark for measuring **Paraphrase Sensitivity Failures (PSFs)** in medical Vision-Language Models, derived from the [VinDR-CXR-VQA](https://huggingface.co/datasets/faizan711/VinDR-CXR-VQA) dataset. ## Overview PSF occurs when a VLM changes its answer to a clinically equivalent question simply because it is worded differently. This dataset provides systematically generated paraphrases across 5 linguistic phenomena, enabling reproducible evaluation of VLM robustness in radiology question-answering. | Statistic | Value | |-----------|-------| | Flip bank rows | 31,444 | | Unique question templates | 42 | | Full image-question pairs | 8,820 | | Unique images | 4,315 | | Pathologies | 14 | | Linguistic phenomena | 5 | ## Files | File | Description | Rows | |------|-------------|------| | `vindr_flip_bank.csv` | Main evaluation set: original-paraphrase pairs with metadata | 31,444 | | `vindr_questions.csv` | Template questions with answer distribution statistics | 42 | | `vindr_questions_full.csv` | All image-question pairs with ground-truth answers and bounding boxes | 8,820 | | `vindr_paraphrases_filtered.json` | Paraphrase generation output with similarity scores and filtering metadata | 146 paraphrases | ## Pathologies (14) Aortic enlargement, Atelectasis, Calcification, Cardiomegaly, Consolidation, ILD, Infiltration, Lung Opacity, Nodule/Mass, Other lesion, Pleural effusion, Pleural thickening, Pneumothorax, Pulmonary fibrosis ## Linguistic Phenomena (5) | Phenomenon | Description | Example | |------------|-------------|---------| | `lexical_substitution` | Synonym or near-synonym replacement | "show" -> "present" | | `syntactic_restructuring` | Reordering clause structure | "Does X show Y?" -> "Is Y shown on X?" | | `negation_pattern` | Introducing or restructuring negation | "show" -> "rule out" | | `scope_quantification` | Changing quantifier scope | "any" -> "all", "some" -> "each" | | `specificity_modulation` | Adjusting specificity level | "opacity" -> "ground-glass opacity" | ## Flip Bank Schema | Column | Type | Description | |--------|------|-------------| | `question_id` | str | Unique identifier (vd-NNNNNN) | | `original_question` | str | Source question from VinDR-CXR-VQA | | `paraphrase` | str | Generated paraphrase | | `phenomenon` | str | Linguistic phenomenon category | | `op_orig` | str | Detected operation of original (presence, location, etc.) | | `op_para` | str | Detected operation of paraphrase | | `op_match` | bool | Whether original and paraphrase have matching operations | | `ground_truth` | str | Expected answer | | `question_type` | str | Question type (Is_there, Where, Yes_No) | | `gt_finding` | str | Ground-truth pathology finding | | `gt_location` | str | Spatial location token | | `bbox` | str | Bounding box coordinates [x1, y1, x2, y2] | | `image_id` | str | Image identifier (DICOM hash) | | `dicom_path` | str | Original DICOM file path | ## Generation Method 1. **Template extraction**: 42 unique question templates from VinDR-CXR-VQA 2. **Paraphrase generation**: GPT-5.4 via OpenAI Batch API, generating 5 paraphrases per question across 5 linguistic phenomena 3. **Semantic filtering**: BioClinicalBERT (`emilyalsentzer/Bio_ClinicalBERT`) embedding similarity filtering with threshold 0.95 (edge cases at 0.90) 4. **Expansion**: Filtered paraphrases crossed with all applicable image-question pairs to produce the 31,444-row flip bank ## Usage ```python from datasets import load_dataset # Load the flip bank ds = load_dataset("saillab/VinDR-CXR-PSF", data_files="vindr_flip_bank.csv") # Or load specific files flip_bank = load_dataset("saillab/VinDR-CXR-PSF", data_files="vindr_flip_bank.csv", split="train") questions = load_dataset("saillab/VinDR-CXR-PSF", data_files="vindr_questions.csv", split="train") ``` ## Images This dataset contains **metadata only** (questions, paraphrases, annotations). Chest X-ray images must be obtained separately from the [VinDR-CXR](https://physionet.org/content/vindr-cxr/1.0.0/) dataset on PhysioNet (requires credentialed access) or from [faizan711/VinDR-CXR-VQA](https://huggingface.co/datasets/faizan711/VinDR-CXR-VQA) on HuggingFace Hub. ## Source Derived from [faizan711/VinDR-CXR-VQA](https://huggingface.co/datasets/faizan711/VinDR-CXR-VQA), which is itself derived from VinDR-CXR (Nguyen et al., 2022). ## Citation If you use this dataset, please cite the PSF-Med benchmark: ```bibtex @inproceedings{sadanadan2026psfmed, title={PSF-Med: Measuring and Mitigating Paraphrase Sensitivity Failures in Medical Vision-Language Models}, author={Sadanadan, Binesh and Behzadan, Vahid}, booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, year={2026} } ``` ## License CC-BY-4.0 (same as the source VinDR-CXR-VQA dataset).
